PetaPixel: This Guy Figured Out How to ‘Cut and Paste’ the Real World Into Photoshop

Artist, designer and programmer Cyril Diagne recently created a bit of tech that looks more like science fiction that science fact. Using a combination of augmented reality and machine learning tech, he’s figured out a way to “copy and paste” objects from the real world into Photoshop, using just a smartphone.

Diagne unveiled his creation over on Twitter earlier this week, where even the Adobe Photoshop account was blown away by the results:

4/10 – Cut & paste your surroundings to Photoshop

Code: https://t.co/cVddH3u3ik

Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF

— Cyril Diagne (@cyrildiagne) May 3, 2020

What might, at first blush, seem like special effects trickery or an April Fools prank is actually a piece of bona-fide tech that he’s actually published to GitHub. As the description on GitHub explains, this is “an AR+ML prototype that allows cutting elements from your surroundings and pasting them in an image editing software.” He’s using Photoshop, but it could, in theory, work with anything.

According to Diagne, the “secret sauce” is a piece of machine learning (AKA “Artificial Intelligence”) tech called BASNet, which automatically recognizes and cuts out real-world objects when taking a photograph. That takes care of the “copy” in “copy and paste.”

BASNet is then combined with another piece of tech called OpenCV SIFT, which allows Diagne’s computer to sync up with his phone and detect where exactly he’s pointing the camera for the “paste” part of this trick.

And again, the OpenCV SIFT trick to find where the phone is pointing at the screen.

I also packaged it as a small python library: https://t.co/en0EyGSklp

Send a camera image + a screenshot and you get accurate x, y screen coordinates! pic.twitter.com/OOCFrrZseZ

— Cyril Diagne (@cyrildiagne) May 3, 2020

All of this comes together into an app that Diagne designed for his smartphone, and the results are nothing short of astounding. It takes about 2.5 seconds for the phone to cut an object out of the real world, and another 4 seconds to paste that object into a Photoshop document.

As we mentioned above, the code behind AR Cut and Paste is available on GitHub, but keep in mind that this is a “research prototype” and not a consumer-ready feature. That said, now that we know Adobe is watching, we sincerely hope they’re planning to integrate something like this into Photoshop ASAP.

from PetaPixel https://ift.tt/3deK9rs
via IFTTT