Boulder Future Salon News Bits

A new way for neural networks to interpret visual data -- using graphs. Graphs in the sense of nodes connected by edges, not like graph paper. The idea is to represent a scene in the form of a graph where the graph is hierarchical and the nodes at the highest level of the graph represent the largest groupings, that is, whole objects, while those near the bottom of the hierarchy represent subparts of the object, and subparts of the subparts, and so on. Edges represent within-object "bonds" that hold the parts together. The actual data content of each node represents meaningful properties of the object, such as its shape, orientation, position, texture, and so on.

The researchers call this a "physical scene graph", abbreviated PSG, and the neural network that they developed to create the physical scene graph is called PSGNet. As you might imagine, getting this to work required some clever tricks. The first was to take a convolutional neural network and add feedback connections to it, both between adjacent layers, and long-range, spanning many layers. Theirs has 5 layers. Since a neural network that has feedback loops is called a recurrent neural network, they call this a convolutional recurrent neural network, or ConvRNN.

The next trick is what they call their graph pooling and graph vectorization operations. The process is started by creating a feature set from the most high-resolution layer of the ConvRNN. Graph vectorization uses features from the ConvRNN combined with existing nodes to produce new nodes with the attributes described above. Graph pooling combines existing nodes. It alternates between graph vectorization and graph pooling and with each cycle through both, steps up to the next higher level in the hierarchy. Edges between layers in the hierarchy are added on the pooling step.

If you're wondering what the rules are for determining what gets made into vectors and then pooled into objects and edges, the next clever trick they came up with is making all of this a system that learns from the data as well. So there are no fixed rules for what in image has "affinity" to be grouped up into objects. It's all learned. It uses a "self-supervised" system. That means it uses the data it has seen so far to predict what will come next. Oh, I probably should have mentioned, this system needs video, not just still photos. Using this idea the system trains an "affinity function" that is mainly determined by motion -- nodes that move together get grouped together.

"Neural Synesthesia is an AI art project that aims to create new and unique audiovisual experiences with artificial intelligence. It does this through collaborations between humans and generative networks. The results feel almost like organic art. Swirls of color and images blend together as faces, scenery, objects, and architecture transform to music. There's a sense of things swinging between feeling unique and at the same time oddly familiar."

"I've always had a fascination for aesthetics. Examples are mountain panoramas, indie game design, scuba diving in coral reefs, psychedelic experiences, and films by Tarkovsky. Beautiful visual scenes have the power to convey meaning without words. It's almost like a primal, visual language we all speak intuitively."

"When I saw the impressive advances in generative models (especially GANs), I started imagining where this could lead. Just like the camera and the projector brought about the film industry, I wondered what narratives could be built on top of the deep learning revolution."

High resolution neural face swapping. Deepfakes taken to the next level. There's one encoder for any input face, but every output face has its own output decoder trained for that face. The decoding has a "progressive" system where it adds resolution is steps, rather than going end-to-end in high resolution. Face alignment by detecting facial landmarks combined with an "ablation" system eliminates any jitter. The background is composited using a separate compositing process.

Japan in 1913-1915 colorized and upscaled by a neural network. Looks pretty good but not quite realistic. The bouncing ball and juggling didn't work. The ambient sound is completely artificial -- the original footage didn't have sound. And apparently in 1913-1915 cameras were a novelty as a lot of people are just staring at the camera.

Price of gold broke $1,800 this morning. [1,802.70 at time of posting]. Looks like inflation is on its way.

The Dark Energy Spectroscopic Instrument, or DESI, has been installed at the Kitt Peak National Observatory in Arizona and will soon come on line. "The device features 5,000 optical fibers, each one designed to collect light from a single galaxy."

"A previous instrument on a different telescope, the Baryon Oscillation Spectroscopic Survey instrument, required collaborators to drill 1,000 holes into large metal plates that held fibers in a configuration that exactly matched the position of known galaxies in a small portion of the night sky. Each time scientists wanted to image new galaxies, a new plate had to be drilled and the fibers inserted by hand."

"With DESI, researchers have relegated the grueling work of pinpointing galaxy locations to a hive of 5,000 robotic pencil-shaped tubes. The positioners have a precision of several micrometers -- about one-10th the width of a human hair -- and are capable of moving on their own to focus on distant galaxies."

"Researchers implemented a software package called Platemaker, which was designed by Kent and scientist Eric Neilsen at Fermilab. The software is a key player in choreographing the movement of all 5,000 robotic positioners simultaneously, especially since the positioners can sometimes get in each other's way."

"First, the focal plane -- a large metallic structure that holds the positioners in place -- must be pointed at just the right portion of sky." "High-resolution cameras embedded in the focal plane capture and analyze light from stars, which allows researchers to orient the telescope."

"Once the telescope is pointed in the right direction, the robotic positioners begin an intricate mechanical waltz, peering deep into the sky to detect sources of light far too faint for human eyes to see."

"Their high degree of precision gets them most of the way to the desired galaxy, but the angle might still be slightly off for some. To get them the rest of the way, DESI has a CCD camera installed at the primary mirror of the telescope, which looks up at the focal plane. Researchers use a built-in light source to illuminate the fibers embedded in the robotic positioners. The fibers project the resulting small dots of light to the CCD camera, which then images them. The Platemaker software compares the positions of the fibers in the images to where they should actually be pointed based on detailed star charts from previous surveys."

"A new study from Penn Medicine lends further evidence that the social behaviors tied to autism spectrum disorders (ASD) emerge from abnormal function of sensory neurons outside the brain." That's interesting as autistic people often have sensitivities to light, sound, touch, and other senses, that non-autistic people don't have.

This experiment was done in fruit flies, though. What, fruit flies can be autistic? Well, what they did was take a human gene, called NF1, and find an analogous gene in the fruit fly Drosophila. They then showed that inhibiting the function of the protein produced by this gene, neurofibromin 1, caused "social impairments," at least in the male flies. If you don't think of fruit flies as being particularly social, well, they are when mating, apparently. Anyway, the researchers showed that the loss of the Nf1 protein caused a type of cell that senses pheromones called ppk23+ to decrease firing. ppk23+ normally inhibits courtship behavior so the disruption of ppk23+ caused increase of behaviors like single-wing extensions at inappropriate times ("when not oriented to the target").

"The brain requires a disproportionate amount of energy compared to its body mass." What does the brain use this energy for? To do this, researchers tracked oxygen consumption. Never mind that it was in tadpoles, rather than humans. It turns out it's easier to keep tadpole brains alive in a special solution in a lab.

Anyway, when at rest, nerve cells use 50% of the oxygen and glial cells use the other 50%. However, when active, nerve cells consume more oxygen. But when the neuron spikes, oxygen can go down as much as another 75% (though usually not that much) from the level it was to start with. This is the oxygen level in the ventricle in the tadpole brain where they were doing the measurements -- the more the tadpole's brain uses the oxygen, the more the oxygen in the ventricle gets used up and that drop can be measured. Firing and then restoring the voltage-gated sodium ion channels that the tadpole's brain uses consumes a lot of energy. This can be verified by using a local anesthetic for fish called tricaine methanesulfonate (MS-222, not to be confused with MS-13). It works by preventing sodium ions from entering cells and thereby prevents the action potentials of the sodium ion channels from being created.

How relationships between different smells are encoded in the olfactory cortex. "To investigate, the researchers developed an approach to quantitatively compare odor chemicals analogous to how differences in wavelength, for example, can be used to quantitatively compare colors of light."

"They used machine learning to look at thousands of chemical structures known to have odors and analyzed thousands of different features for each structure, such as the number of atoms, molecular weight, electrochemical properties and more. Together, these data allowed the researchers to systematically compute how similar or different any odor was relative to another."

"They then exposed mice to various combinations of odors from the different sets and used multiphoton microscopy to image patterns of neural activity in the piriform cortex and olfactory bulb."

"The experiments revealed that similarities in odor chemistry were mirrored by similarities in neural activity. Related odors produced correlated neuronal patterns in both the piriform cortex and olfactory bulb, as measured by overlaps in neuron activity."

"In the cortex, related odors led to more strongly clustered patterns of neural activity compared with patterns in the olfactory bulb. This observation held true across individual mice. Cortical representations of odor relationships were so well-correlated that they could be used to predict the identity of a held-out odor in one mouse based on measurements made in a different mouse."

"Additional analyses identified a diverse array of chemical features, such as molecular weight and certain electrochemical properties, that were linked to patterns of neural activity. Information gleaned from these features was robust enough to predict cortical responses to an odor in one animal based on experiments with a separate set of odors in a different animal."

AAVE stands for African American Vernacular English. Linguists who have studied AAVE have found it is not an inferior or incorrect version of standard English, but a full, grammatically consistent dialect of English.

"New genomic atlas of the developing human brain." "The researchers studied cells from a section of the developing human brain called the telencephalon. This region contains structures responsible for sensory processing, voluntary movement, language, and communication."

"The team took advantage of the fact that inside cells, the genome is tightly wound into a dense structure known as chromatin. This three-dimensional structure reveals the important parts of the genome in any given cell by exposing the stretches of regulatory DNA needed for the cell to function. Using a technology called ATAC-seq, the team cut up exposed DNA in embryonic brain cells. By analyzing where these cuts are made, they were able to surmise what parts of the genome are exposed and might contain important regulatory regions."

"Their initial experiments revealed more than 103,000 regions of open chromatin in the developing brain cells. To narrow down that list, the researchers turned to a machine-learning approach. They wrote a computer program that uses information already known about regulatory DNA to help pick out patterns specific to brain cells."

"If a regulatory region was similar to one known to only be active in limbs or lungs, for instance, the machine-learning program concluded that it wasn't a brain-specific enhancer. In the end, the group came up with a set of about 19,000 regulatory regions of the genome expected to play a role in brain development."

The machine learning algorithms used were elastic net and random forest classifiers. I've described random forest before, so won't repeat that here except to say it basically makes many decision "trees" (hence "forest") which are combined together as an ensemble to produce the result. I've also described the "Lasso" method of reducing the number of parameters in a linear regression model. "Elastic net" combines the "lasso" method with a "ridge" method that is also designed for doing linear regression with a very large number of parameters. I'm going to skip the details because it's all done using matrices and not simple to summarize in words.

Anyway, the machine learning models were trained on and generated predictions for open chromatin regions for all brain regions. Training labels were generated using open chromatin regions in combination with an existing enhancer database. Open chromatin regions were labeled positive or negative if they matched validated brain enhancers or not. Open chromatin regions were considered predictive regulatory elements if either algorithm's prediction was above 0.5. That how they got the 19,151 out of 103,829 open chromatin regions.

Although the article only mentions human brain cells, the researchers actually used both transgenic mice and human brain-derived neuroblastoma cells. They also used CRISPR and lentivirus to extract RNA and reverse transcribe to DNA, as well as a variety of statistical tools to identify regulatory elements before feeding the data into the machine learning algorithm, and after that to find genetic sequences that bind to transcription factors (which are called motifs). A transcription factor is a protein that controls the rate of transcription of genetic information from DNA to messenger RNA, by binding to a specific DNA sequence. The idea is to make sure that genes are expressed in the right cell at the right time.

"How does our brain fold? Study reveals new genetic insights." "New research is helping unlock the mystery of how the brain folds as a baby develops in the womb -- a process critical to healthy brain function. Misfolding of the brain is linked with many neurological conditions including autism, anorexia, epilepsy and schizophrenia."

Never mind that the picture is of a human baby, but the research was actually done with fetal sheep.

"Grey matter is made up of neuron bodies and their connecting arms, while white matter is composed of the neurons' long nerve fibres and their protective layer of fat."

"While the science of folding is still unclear, the latest evidence suggests grey matter in the developing brain expands faster than white matter, creating mechanical instability that leads to brain folding."

"But the resulting 'hill' and 'valley' folds are not random -- they follow a similar pattern in all folded brains of the same species."

"Researchers investigated the genetic and microstructural differences in future grey matter, the cortical plate, in the parts of the brain just beneath the 'hills' and 'valleys'."

"The areas were analysed at three points of development -- when the brain was smooth, semi-folded and fully folded."

"We found some genes have higher expression in regions that fold outward and lower expression in regions that fold inwards. Other genes reverse this pattern."

"These genetic differences are also correlated with changes in grey matter neurons, with the study finding variations in the number of arms -- or dendrites -- that neurons grow in these regions during the folding process."

More specifically, the genes that had higher gene expression in cells in regions that became "hills" (gyri) were BDNF, CDK5, and NeuroD6, while the genes that had higher gene expression in regions that became "valleys" (sulci) were HDAC5 and MeCP2.

BDNF is a gene that encodes for brain-derived neurotrophic factor, which is involved in growth and differentiation of nerve cells and also their synapses. CDK5 is a gene that encodes for cell division protein kinase 5, which in turn plays a role in regulating the cell cycle, regulating the process of mRNA processing and transcribing DNA into proteins, and the differentiation of nerve cells. NeuroD6 is a gene that encodes for neuronal differentiation 6, a protein known to be associated with epilepsy.

HDAC5 is a gene that encodes for histone deacetylase 5, which is an enzyme that plays a role in regulating DNA transcription and cell cycle progression by altering chromosome structure and regulating transcription factor access to DNA. MECP2 is a gene that encodes for methyl CpG binding protein 2, a protein present in high levels in nerve cells after they reach maturity, which turns off several other genes according to DNA methylation.

"TikTok is a data collection service that is thinly-veiled as a social network. If there is an API to get information on you, your contacts, or your device... well, they're using it." Phone hardware, other apps installed, everything network related, whether you're rooted/jailbroken, your GPS, ... "They encrypt all of the analytics requests with an algorithm that changes with every update (at the very least the keys change) just so you can't see what they're doing." "For what it's worth, I've reverse the Instagram, Facebook, Reddit, and Twitter apps. They don't collect anywhere near the same amount of data that TikTok does, and they sure as hell aren't outright trying to hide exactly what's being sent like TikTok is."

"To fill the gap between realistic but infeasible real-world tasks, and the somewhat lacking but easy-to-use simulated tasks, we recently introduced the D4RL benchmark (Datasets for Deep Data-Driven Reinforcement Learning) for offline RL. The goal of D4RL is simple: we propose tasks that are designed to exercise dimensions of the offline RL problem which may make real-world application difficult, while keeping the entire benchmark in simulated domains that allow any researcher around the world to efficiently evaluate their method. In total, the D4RL benchmark includes over 40 tasks across 7 qualitatively distinct domains that cover application areas such as robotic manipulation, navigation, and autonomous driving."

"We begin with 3 navigation domains of increasing difficulty. The easiest of the domains is the Maze2D domain, which tries to navigate a ball along a 2D plane to a target goal location."

"For a realistic, vision-based navigation task, we use the CARLA simulator. This task adds a layer of perceptual challenge on top of the two aforementioned Maze2D and AntMaze tasks."

"We also include 2 realistic robotic manipulation tasks, using the Adroit (based on the Shadow Hand robot) and the Franka platforms. The Adroit domain contains 4 separate manipulation tasks, as well as human demonstrations recorded via motion capture. This provides a platform for studying the use of human-generated data within a simulated robotic platform."

"The Franka kitchen environment places a robot in a realistic kitchen environment where objects can be freely interacted with. These include opening the microwave and various cabinets, moving the kettle, and turning on the lights and burners."

"We include two tasks from the Flow benchmark (which has been covered in a previous blog post). The Flow project proposes to use autonomous vehicles for reducing traffic congestion, which we believe is a compelling use case for offline RL. We include a ring layout and a highway merge layout."

"Finally, we also include datasets for HalfCheetah, Hopper, and Walker2D from the OpenAI Gym Mujoco benchmark."

Results from a universal basic income experiment done in Finland in 2017 and 2018 have been released. "In the experiment, a sum of 560 euros per month was paid to a randomly selected group of 2,000 unemployed Finns on a flat rate unemployment benefit. The basic income replaced the existing unemployment benefits and was paid even if the participants took up jobs. The rest of the unemployed in Finland who receive a flat rate unemployment benefit, formed the control group. Since the group getting the 'treatment', i.e. the 560 euros a month net sum, was similar to the control group in all relevant background characteristics, the experiment mimicked studies in natural sciences and medicine. The idea was that if there were any differences between the treatment group and the control group after the experiment, we would be able to establish a causal loop."

The results are described as "dis­ap­point­ing" because it was hoped the universal basic income group would have higher employment. In fact it was about the same as the control group. At least it didn't go down. The researchers have more confidence in the results for the first year, because in the second year, the Finnish government started some "activation" program that affected everyone in both groups. The article does not clarify what this "activation" program was other than saying it had more stringent entitlement criteria for unemployment.

People in the universal basic income experiment reported less stress and greater wellbeing, but "the government was not primarily interested in improving the wellbeing of unemployment benefit claimants." The government only cared whether "improving financial incentives would lead to increased employment."

COVID-19 Earth Observation Dashboard. To learn how to use it, watch the tutorial video (below). The site itself is at . Shows how the coronavirus pandemic impacted life on the ground as visible from space. You can see imports and production, airport and ship throughput, light from cities at night, air quality, and various other indicators. The right panel will have a thorough description explaining the data that you are seeing and what satellite it came from. The satellites are operated by NASA, ESA, and JAXA (Japan's space agency). The left panel lets you choose what indicator you want to look at, the center panel has a map, and you can click the map to change what gets shown in the right panel. Sometimes you can click things in the right panel that will overlay data back on to the map.