R&D Roundup: Ultrasound/AI medical imaging, assistive exoskeletons and neural weather modeling

In the time of COVID-19, much of what transpires from the science world to the general public relates to the virus, and understandably so. But other domains, even within medical research, are still active — and as usual, there are tons of interesting (and heartening) stories out there that shouldn’t be lost in the furious activity of coronavirus coverage. This last week brought good news for several medical conditions as well as some innovations that could improve weather reporting and maybe save a few lives in Cambodia.

Ultrasound and AI promise better diagnosis of arrhythmia

Arrhythmia is a relatively common condition in which the heart beats at an abnormal rate, causing a variety of effects, including, potentially, death. Detecting it is done using an electrocardiogram, and while the technique is sound and widely used, it has its limitations: first, it relies heavily on an expert interpreting the signal, and second, even an expert’s diagnosis doesn’t give a good idea of what the issue looks like in that particular heart. Knowing exactly where the flaw is makes treatment much easier.

Ultrasound is used for internal imaging in lots of ways, but two recent studies establish it as perhaps the next major step in arrhythmia treatment. Researchers at Columbia University used a form of ultrasound monitoring called Electromechanical Wave Imaging to create 3D animations of the patient’s heart as it beat, which helped specialists predict 96% of arrhythmia locations compared with 71% when using the ECG. The two could be used together to provide a more accurate picture of the heart’s condition before undergoing treatment.

Another approach from Stanford applies deep learning techniques to ultrasound imagery and shows that an AI agent can recognize the parts of the heart and record the efficiency with which it is moving blood with accuracy comparable to experts. As with other medical imagery AIs, this isn’t about replacing a doctor but augmenting them; an automated system can help triage and prioritize effectively, suggest things the doctor might have missed or provide an impartial concurrence with their opinion. The code and data set of EchoNet are available for download and inspection.

You can now buy AWS’ $99 DeepComposer keyboard

AWS today announced that its DeepComposer keyboard is now available for purchase. And no, DeepComposer isn’t a mechanical keyboard for hackers but a small MIDI keyboard for working with the AWS DeepComposer service that uses AI to create songs based on your input.

First announced at AWS re:Invent 2019, the keyboard created a bit of confusion, in part because Amazon’s announcement almost made it seem like a consumer product.

DeepComposer, which also works without the actual hardware keyboard, is more of a learning tool, though, and belongs to the same family of AWS hardware like DeepLens and DeepRacer. It’s meant to teach developers about generative adversarial networks, just like DeepLens and DeepRacer also focus on specific machine learning technologies.

Users play a short melody, either using the hardware keyboard or an on-screen one, and the service then automatically generates a backing track based on your choice of musical style.

The results I heard at re:Invent last year were a bit uneven (or worse), but that may have improved by now. But this isn’t a tool for creating the next Top 40 song. It’s simply a learning tool. I’m not sure you need the keyboard to get that learning experience out of it, but if you do, you can now head over to Amazon and buy it.

 

Microsoft brings Teams to consumers and launches Microsoft 365 personal and family plans

Microsoft today announced a slew of new products, but at the core of the release is a major change to how the company is marketing its tools and services to consumers.

Office 365, which has long been the brand for the company’s subscription service for its productivity tools like Word, Excel and Outlook, is going away. On April 21, it’ll be replaced by new Microsoft 365 plans, including new personal and family plans (for up to six people) at $6.99 and $9.99 respectively. That’s the same price as the existing Office 365 Personal and Home plans.

“We are basically evolving our subscription from — in our minds — a set of tools to solutions that help you manage across your work and life,” Yusuf Mehdi, Microsoft’s CVP of Modern Life, Search and Devices, told me ahead of today’s announcement.

Microsoft is making similar branding changes to its business plans for Office 365. They are a bit more convoluted, with Office 365 Business Premium now called Microsoft 365 Business Standard and Microsoft 365 Business now becoming Microsoft 365 Business Premium, but for the most part, this is about branding while prices stay the same.

These new Microsoft 365 Personal and Family plans will include access to Outlook and the Office desktop apps for Windows and macOS, 1 terabyte of OneDrive storage per person (including unlimited access to the more secure OneDrive Personal Vault service) and 50 gigabytes of Outlook.com email storage, Skype call recording and 60 minutes of Skype landline and mobile phone calls.

And since this is now Microsoft 365 and not Office 365, you can also get Windows 10 technical support with the subscription, as well as additional security features to protect you from phishing and malware attacks.

More than 37 million people currently have personal Office 365 subscriptions and chances are these lower prices will bring more users to the platform in the long run. As Mehdi stressed, Microsoft’s free offerings aren’t going away.

But with today’s release, Microsoft isn’t just changing the branding and launching these new plans, it’s also highlighting quite a few new capabilities in its various applications that are either launching today or in the coming months.

Microsoft Teams gets a personal edition

The highlight of this launch, especially given the current situation around COVID-19, is likely the announcement of Teams for consumers. Teams is already one of Microsoft’s fastest growing products for businesses with 44 million people using it. But in its efforts to help people bridge their work and personal lives, it will now launch a new Teams edition for consumers, as well.

Just like you can switch between work and personal accounts in Outlook, you will soon be able to do the same in Teams. The personal teams view will look a little bit different, with shared calendars for the family, access to OneDrive vaults, photo sharing, etc., but it sits on the same codebase as the business version. You’ll also be able to do video calls and shared to-do lists.

Better writing through AI

About a year ago, Microsoft announced that Word Online would get a new AI-powered editor that would help you write better. You can think of it as a smarter grammar checker that can fix all of your standard grammar mistakes but can also help you avoid overly complex sentences and bias in your word choices.

This editor is now the Microsoft Editor, and the company is expanding it well beyond Word. The new AI-powered service is now available in 20 languages in Word and Outlook.com — and maybe most importantly, it’ll be available as a Microsoft Edge and Google Chrome plug-in, too.

Free users will get basic spelling and grammar features, while Microsoft 365 subscribers will get a number of more advanced features like the ability to ask the editor to suggest a rewrite of a mangled sentence, a plagiarism checker, style analysis to see if your writing is unclear or too formal and access to an inclusive language critique to help you avoid unintentional bias.

If you’ve used Grammarly in the past, then a lot of this will sound familiar. Both services now offer a similar set of capabilities, but Microsoft may have an edge with its ability to rewrite sentences.

Better presentations through technology

In a similar vein, Microsoft also launched a presentation coach for PowerPoint as a limited test last September. This AI-driven feature helps you avoid filler words and other presentation no-nos.

This feature first launched in the online version of PowerPoint, with a basic set of features. Now, Microsoft 365 subscribers will get two new advanced features, too, that can help you avoid a monotone pitch that puts your audience to sleep and avoid grammar mistakes in your spoken sentences.

Currently, these are still available as a free preview to all but will become Microsoft 365-only features soon.

PowerPoint is also getting an updated Designer to help you create better presentations. It can now easily turn text into a timeline, for example, and when you add an image, it can present you with a set of potential slide layouts.

Microsoft 365 subscribers now also get access to over 8,000 images and 175 looping videos from Getty Images, as well as 300 new fonts and 2,800 new icons.

Excel + Plaid

For you spreadsheet jockeys out there, Microsoft also has some good news, especially if you want to use Excel to manage your personal budgets.

In partnership with Plaid, you can now link your bank accounts to Excel and import all of your expenses into your spreadsheets. With that, you can then categorize your spend and build your own personal Mint. This feature, dubbed “Money in Excel,” will launch in the U.S. in the coming months.

In addition, Excel is getting a lot more cloud- and AI-driven data types that now cover over 100 topics, including nutrition, movies, places, chemistry and — because why not — Pokémon. Like some of the previous features, this is an extension of the work Microsoft did on Excel in the last few years, starting with the ability to pull in stock market and geographical data.

And just like with the previous set of features, you’ll need a Microsoft 365 subscription to get access to these additional data types. Otherwise, you’ll remain restricted to the stock market and geography data types, which will become available to Office Insiders in the spring and then Personal and Family subscribers in the U.S. in the coming months.

Outlook gets more personal

Even though you may want to forget about Outlook and ignore your inbox for a while, Microsoft doesn’t. In Outlook on the web, you can now link your personal and work calendars to ensure you don’t end up with a work meeting in the middle of a personal appointment, because Chris from marketing really needs another sync meeting during your gym time even though a short email would suffice.

Outlook for Android can now summarize and read your emails aloud for you, too. This feature will roll out in the coming months.

Family Safety

While most of the new features here focus on existing applications, Microsoft is also launching one completely new app: Microsoft Family Safety. This app is coming to Microsoft 365 subscribers on iOS and Android and will bring together a set of tools that can help families manage their online activities and track the location of family members.

The app lets families manage the screen time of their kids (and maybe parents, too) across Windows, Android and Xbox, for example. Parents can also set content filters that only allow kids to download age-appropriate apps. But it also allows parents to track their kids in the real world through location tracking and even driving reports. This, as Mehdi stressed, is a feature that kids can turn off, but they’ll probably have to explain themselves to their parents then. Indeed, he stressed that a lot of what the app does is give parents a chance to have a dialog with their kids. What makes the service unique is that it works across platforms, with iOS support coming in the future.

This app is launching as a limited preview now and will be available in the coming months (I think you can spot a trend here).

Partner benefits

Mehdi noted that Microsoft is also partnering with companies like Adobe, Bark, Blinkist, Creative Live, Experian, Headspace and TeamSnap to provide Microsoft 365 subscribers with additional benefits like limited-time access to their products and services. Subscribers will get three months of free access to Adobe’s Creative Cloud Photography plan, for example.

At the core of today’s updates, though, is a mission to bring a lot of the productivity tools that people know from their work life to their personal life, too, with the personal edition of Teams being the core example.

“We’re very much excited to bring this type of value — not increase the price of Office 365 — take a big step forward, and then move to this,” Mehdi said. “We think now more than ever, it is valuable for people to have the subscription service for their life that helps them make the most of their time, protects their family, lets them develop and grow. And our goal or aspiration is: Can we give you the most valuable subscription for your life? I know people value their video subscriptions and music subscriptions. Our aspiration is to provide the most valuable subscription for your life via Microsoft 365 Personal and Family.”

FluSense system tracks sickness trends by autonomously monitoring public spaces

One of the obstacles to accurately estimating the prevalence of sickness in the general population is that most of our data comes from hospitals, not the 99.9 percent of the world that isn’t hospitals. FluSense is an autonomous, privacy-respecting system that counts the people and coughs in public spaces to keep health authorities informed.

Every year has a flu and cold season, of course, though this year’s is of course far more dire. But it’s like an ordinary flu season in that the main way anyone estimates how many people are sick is by analyzing stats from hospitals and clinics. Patients reporting “influenza-like illness” or certain symptoms get aggregated and tracked centrally. But what about the many folks who just stay home, or go to work sick?

We don’t know what we don’t know here, and that makes estimates of sickness trends — which inform things like vaccine production and hospital staffing — less reliable than they could be. Not only that, but it likely produces biases: Who is less likely to go to a hospital, and more likely to have to work sick? Folks with low incomes and no healthcare.

Researchers at the University of Massachusetts Amherst are attempting to alleviate this data problem with an automated system they call FluSense, which monitors public spaces, counting the people in them and listening for coughing. A few of these strategically placed in a city could give a great deal of valuable data and insight into flu-like illness in the general population.

Tauhidur Rahman and Forsad Al Hossain describe the system in a recent paper published in an ACM journal. FluSense basically consists of a thermal camera, a microphone, and a compact computing system loaded with a machine learning model trained to detect people and the sounds of coughing.

To be clear at the outset, this isn’t recording or recognizing individual faces; Like a camera doing face detection in order to set focus, this system only sees that a face and body exists and uses that to create a count of people in view. The number of coughs detected is compared to the number of people, and a few other metrics like sneezes and amount of speech, to produce a sort of sickness index — think of it as coughs per person per minute.

A sample setup, above, the FluSense prototype hardware, center, and sample output from the thermal camera with individuals being counted and outlined.

Sure, it’s a relatively simple measurement, but there’s nothing like this out there, even in places like clinic waiting rooms where sick people congregate; Admissions staff aren’t keeping a running tally of coughs for daily reporting. One can imagine not only characterizing the types of coughs, but visual markers like how closely packed people are, and location information like sickness indicators in one part of a city versus another.

“We believe that FluSense has the potential to expand the arsenal of health surveillance tools used to forecast seasonal flu and other viral respiratory outbreaks, such as the COVID-19 pandemic or SARS,” Rahman told TechCrunch. “By understanding the ebb and flow of the symptoms dynamics across different locations, we can have a better understanding of the severity of a novel infectious disease and that way we can enforce targeted public health intervention such as social distancing or vaccination.”

Obviously privacy is an important consideration with something like this, and Rahman explained that was partly why they decided to build their own hardware, since as some may have realized already, this is a system that’s possible (though not trivial) to integrate into existing camera systems.

“The researchers canvassed opinions from clinical care staff and the university ethical review committee to ensure the sensor platform was acceptable and well-aligned with patient protection considerations,” he said. “All persons discussed major hesitations about collection any high-resolution visual imagery in patient areas.”

Similarly, the speech classifier was built specifically to not retain any speech data beyond that someone spoke — can’t leak sensitive data if you never collect any.

The plan for now is to deploy FluSense “in several large public spaces,” one presumes on the UMass campus in order to diversify their data. “We are also looking for funding to run a large-scale multi-city trial,” Rahman said.

In time this could be integrated with other first- and second-hand metrics used in forecasting flu cases. It may not be in time to help much with controlling COVID-19, but it could very well help health authorities plan better for the next flu season, something that could potentially save lives.

Visual One smartens up home security cameras with object and action recognition

“Smart” cameras are to be found in millions of homes, but the truth is they’re not all that smart. Facial recognition and motion detection are their main tricks… but what if you want to know if the dog jumped on the couch, or if your toddler is playing with the stove? Visual One equips cameras with the intellect to understand a bit more of the world and give you more granular — and important — information.

Founder Mohammad Rafiee said that the idea came to him after he got a puppy (Zula) and was dissatisfied with the options he had for monitoring her activities while he was away. Here she is doing what dogs do best:

There are no bad dogs, but chairs are for people

“There were specific things I wanted to know were happening, like I wanted to check if the dog got picked up by the dog walker. The cameras’ motion detection is useless — she’s always moving,” he lamented. “In fact, with a lot of these cameras, just a change in the lighting or wind or rain can trigger the motion alert, so it’s completely impractical.”

“My background is in machine learning. I was thinking about it, and realized we’re at a stage where this problem is starting to become solvable,” he continued.

Some tasks in computer vision, indeed, are as good as solved — detecting faces and common objects such as cars and bikes can be done quickly and efficiently. But that’s not always useful — what’s the point of knowing someone rode their bike past your house? In order for this to have value, the objects need to be understood as part of a greater context, and that’s what Rafiee and Visual One are undertaking.

Unfortunately, it’s far from easy — or else everyone would be doing it already. Identifying a cat is simple, and identifying a table is simple, but identifying a cat on a table is surprisingly hard.

“It’s a very difficult problem. So we’re breaking it down to things we can solve right now, then building on that,” Rafiee explained. “With deep learning techniques we can identify different objects, and we build models on top of those to specify different interactions, or specific objects being in specific locations. Like a car in the wrong spot, or a dog getting on a couch. We can recognize that with high accuracy right now — we have a list of supported objects and models that we’re expanding.”

In case you’re not convinced that the capabilities are that much advanced from the usual “activity in the living room” or “Kendra is at the front door” notifications, here are a few situations that Visual One is set up to detect:

  • Kid playing with the stove
  • Toddler climbing furniture
  • Kid holding a knife
  • Baby left alone for too long
  • Raccoon getting into garbage
  • Elderly person taking her medications
  • Elderly person in bed for too long
  • Car parked in the wrong spot
  • Garage door left open
  • Dog chewing on a shoe
  • Cat scratching the furniture

The process for creating these triggers is pretty straightforward

If one of those doesn’t make you think “actually… that would be really good to know,” then perhaps a basic security camera is enough for your purposes after all. Not everyone has a knife-curious toddler. But those of you who do are probably scrolling furiously past this paragraph looking for where to buy one of these things.

Unfortunately Visual One isn’t something you can just install on any old existing system — with the prominent exception of Nest, into which it can plug. Camera workflows are generally too locked down for security and privacy purposes to allow for third-party apps and services to be slipped in. But the company isn’t trying to bankrupt everyone with an ultra-luxury offering. It’s using off-the-shelf cameras from Wyze and loading them with its own software stack.

Rafiee said he pictures Visual One as a mid-tier option for people who want to have more than a basic camera setup but aren’t convinced by the more expensive plays. That way the company avoids going head-on with commodity hardware’s race to the bottom or the brand warfare taking place between Google and Amazon’s Nest and Ring. Cameras cost $30-$40, and the service is $7 per month currently.

Ultimately the low-end companies may want to license from Visual One, while the high-end companies will be developing their own full stack at great cost, making it difficult for them to go downmarket. “Hardware is hard, and AI is specialized — unless you’re a giant company it’s hard to do both. I think we can fill the gap in the market for mid-market companies without those resources,” he said.

Of course privacy is paramount as well, and Rafiee said that because of the way their system works, although the AI lives in the cloud and therefore requires the cameras to be online (like most others), no important user data needs to be or will be stored on Visual One servers. “We do inference in the cloud so we can be hardware agnostic, but we don’t need to store any data. So we don’t add any risk,” he said.

Visual One is launching today (after a stint in YC’s latest cohort) with an initial set of objects and interactions, and will continue developing more as it observes which use cases prove popular and effective.