Thoughts on AI replacing coders by 2040

This blog post looks into which methods and technologies that can potentially lead to the replacement of coders in the future, some are of futuristic nature but some are more “low-hanging” wrt automation of (parts of) coding.

The background for this blog post is that researchers Jay Jay Billings, Alexander J. McCaskey, Geoffroy Vallee, and Greg Watson at Oak Ridge Laboratory wrote a paper: Will humans even write code in 2040 and what would that mean for extreme heterogeneity in computing?

Also related to this is (Tesla AI director) Andrej Karpathy’s article: Software 2.0 where he looks back at how AI (primarily Deep Learning) has replaced custom methods for e.g. image and speech recognition, machine translation (++) and generalizes how Deep Learning can further replace a lot of other software in years to come (note: examplified by Google’s recent paper The Case for Learned Index Structures)

1. FACT: Programming environments (IDEs) have barely changed the last 30 years

One of primary purposes of programming is to provide efficient automation, however programming itself is still a highly manual and labour intensive process – except for refactoring the difference between modern IDEs compared to e.g. Turbo Pascal in 1989 is surprisingly small? (Turbo Pascal came out more than 30 years ago and improved gradually towards 1989)

Turbo Pascal 5.5 in 1989 Eclipse in 2017

2. FACT: For (close to) all FUNCTIONS written there exists one or several TESTS for it

For any method already written (or to be written) in any (of the most popular) languages currently used in programming  there already exists a test for it – in the same language or in a similar language (e.g. a C# test for a Java function). The obvious (big) data source for this is all private and public repositories in Github  (100M pull requests merged so far in 2017)

So why are most developers still writing unit tests instead of having an IDE/service find and translate the appropriate tests to the functions they write? (e.g. something along the lines of IntelliTest)

3. FACT: as 2 – For (close to) all TESTS there exists a (set of ) FUNCTIONs they test

Assuming e.g. with Test Driven Development (TDD) – where you are roughly writing the (test of the new) API first – and then alternating between creating in the code to (just enough) fulfill the API.

This seems like it has 2 potential ways of being increasingly automated – on the function writing part.

  1. Search for a set of function that matches a set of tests – instead of writing the functions just write the tests
  2. Automatically generate the code (fragments) to fulfill the test, this can potentially be done in many ways, e.g.
      1. Brute force with a lot of computers (e.g. a few thousand CPUs in the cloud should be more than capable of quickly generating and selecting the best of maybe up to 30-50 increments needed per test writing iteration, this resource could be shared by a large set of programmers). See also the science of brute force.
      2. Using sophisticated AI methods with GPUs – e.g.  Program Synthesis with Deep Learning and earlier AI methods such as variants of John Koza’s Genetic Programming.
      3. Quantum Computer – e.g. Logic Synthesis with Quantum Computing (see also Quantum Development Kit)

4. FACT: (Many?) Data Structures can now be replaced by AI (e.g. Bloom Filters & B-Trees)

Google – with Tim KraskaAlex BeutelEd H. ChiJeffrey DeanNeoklis Polyzotis published an interesting paper: The Case for Learned Index Structures where they showed that traditional and popular data structures such as B-Tree, Hash Index and Bloom Filter can be with advantage be replaced by AI Trained Index Structures. This is from my perspective pretty groundbreaking research and will be interesting to see what the use cases can be towards program synthesis (e.g for other categories of data structures and also logic operating on data structures). Key result figures from their paper:

 

5. FACT: Formal Methods are starting to work and can be used to support automation of code

Amazon has used formal methods since 2011 for Amazon Web Services, e.g. to ensure the quality of AWS S3 – see Use of Formal Methods at Amazon for an overview. Facebook is using formal methods for mobile application quality assurance, see Moving Fast with Software Verification. Formal methods has also been used to validate various blockchain technologies (potentially Bitcoin), see e.g. A Temporal Blockchain: A Formal Analysis, Blockchain Protocol Analysis and Security Engineering 2017, Validation of Decentralised Smart Contracts Through Game Theory and Formal Methods

A few years back I participated in a research project – Envisage (Engineering Virtualized Services) – where formal methods were used to prove that the TimSort algorithm was buggy in a lot of languages  (See Hacker News story below). Formal methods can potentially be used together with code generating methods to ensure that what is generated is correct.

Conclusion

Have presented various technologies that might play a role in automating coding forwards. A potential interesting direction on the generative side for coding is to use sequence to sequence deep learning in combination with GAN for synthesis, see e.g. Text2Action: Generative Adversarial Synthesis from Language to Action – for program synthesis this looks like a

Best regards,

Amund Tveit

 

Continue Reading

New Publications in Deep Learning Publication Navigator

Long overdue update of new publications in Deep Learning Publication Navigator (ai.amundtveit.com) – for now the easiest way to discover new publications is probably to convert screenshots (number of papers) per category in the before and after update screenshots below.

Examples of keywords (from publication title) with (several) new Deep Learning publications are:

  1. 3D
  2. Acoustic
  3. Active learning
  4. Adaptive
  5. Adversarial (123 new papers since last update, due to significant activity in GAN Research)
  6. Alzheimer’s (22 new papers related to a disease that cost more than a quarter trillion US$ annually to treat in the USA)
  7. Anomaly detection
  8. Autoencoders
  9. Bayesian
  10. Biomedical
  11. Chinese
  12. Clinical
  13. Collaborative filtering (e.g. for recommender systems)
  14. Dataset
  15. EEG (electric brain signals)
  16. Ensemble
  17. +++++ (many more!)

If you have feature ideas or other requests for Deep Learning Publication Navigator, feel free to reach out.

Best regards,

Amund Tveit

After update (with new papers):

Before update (without new papers):

 

Continue Reading

Creative AI on the iPhone (with GAN) and Dynamic Loading of CoreML models

Zedge summer interns developed a very cool app using ARKit and CoreML (on iOS11). As parts of their journey the published 2 blog posts on the Zedge corporate web site related to:

  1. How to develop and run Generative Adversarial Networks (GAN) for Creative AI on the iPhone using Apple’s CoreML tools, check out their blog post about it.
  2. Deep Learning models (e.g. for GAN) can take a lot of space on a mobile device (tens of Megabytes to perhaps even Gigabytes), in order to keep initial app download size relatively low it can be useful to dynamically load only the models you need. Check out their blog post about various approaches for hotswapping CoreML models.

Best regards,

Amund Tveit 

Continue Reading

Convolutional Neural Networks for Self-Driving Cars

This blog post are my notes from project 3 in the term 1 of the Udacity Nanodegree in Self Driving cars. The project is about developing and training a convolutional neural network of camera input (3 different camera angles) from a simulated car.

Best regards,

Amund Tveit

1. Modelling Convolutional Neural Network for Self Driving Car

Used the NVIDIA Autopilot Deep Learning model for self-driving as inspiration (ref: paper “End to End Learning for Self-Driving Cars” – https://arxiv.org/abs/1604.07316 and implementation of it: https://github.com/0bserver07/Nvidia-Autopilot-Keras), but did some changes to it:

  1. Added normalization in the model itself (ref Lambda(lambda x: x/255.0 – 0.5, input_shape=img_input_shape)), since it is likely to be faster than doing it in pure Python.
  2. Added Max Pooling after the first convolution layers, i.e. making the model a more “traditional” conv.net wrt being capable of detecting low level features such as edges (similar to classic networks such as LeNet).
  3. Added Batch Normalization in early layers to be more robust wrt different learning rates
  4. Used he_normal normalization (truncated normal distribution) since this type of normalization with TensorFlow has earlier mattered a lot
  5. Used L2 regularizer (ref: “rule of thumb” – https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization-How-does-it-solve-the-problem-of-overfitting-Which-regularizer-to-use-and-when )
  6. Made the model (much) smaller by reducing the fully connected layers (got problems running larger model on 1070 card, but in retrospect it was not the model size but my misunderstandings of Keras 2 that caused this trouble)
  7. Used selu (ref: paper “Self-Normalizing Neural Networks” https://arxiv.org/abs/1706.02515) instead of relu as rectifier functions in later layers (fully connected) – since previous experience have shown (with traffic sign classification and tensorflow) showed that using selu gave faster convergence rates (though not better final result).
  8. Used dropout in later layers to avoid overfitting
  9. Used l1 regularization on the final layer, since I’ve seen that it is good for regression problems (better than l2)

Image of Model model Image

Detailed model

####2. Attempts to reduce overfitting in the model

The model contains dropout layers in order to reduce overfitting (ref dropout_1 and dropout_2 in figure above and train_car_to_drive.ipynb).

Partially related: Used also balancing of data sets in the generator, see sample_weight in generator function and snippet below

The model was tested by running it through the simulator and ensuring that the vehicle could stay on the track. See modelthatworked.mp4 file in this github repository.

####3. Model parameter tuning

The model used an adam optimizer, so the learning rate was not tuned manually

####4. Appropriate training data

Used the training data that was provided as part of the project, and in addition added two runs of data to avoid problems (e.g. curve without lane line on the right side – until the bridge started and also a separate training set driving on the bridge). Data is available on https://amundtveit.com/DATA0.tgz).

###Model Architecture and Training Strategy

####1. Solution Design Approach

The overall strategy for deriving a model architecture was to use a conv.net, first tried the previous one I used for Traffic Sign detection based on LeNet, but it didn’t work (probably too big images as input), and then started with the Nvidia model (see above for details about changes to it).

In order to gauge how well the model was working, I split my image and steering angle data into a training and validation set. Primary finding was that numerical performance of the models I tried was not a good predictor of how well it would it perform on actual driving in the simulator. Perhaps overfitting could be good for this task (i.e. memorize track), but I attempted to get a correctly trained model without overfitting (ref. dropout/selu and batch normalization). There were many failed runs before the car actually could drive around the first track.

 

2. Creation of the Training Set & Training Process

I redrove and captured training data for the sections that were problematic (as mentioned the curve without lane lines on right and the bridge and part just before bridge). Regarding center-driving I didn’t get much success adding data for that, but perhaps my rebalancing (ref. generator output above) actually was counter-productive?

For each example line in the training data I generated 6 variants (for data augmentetation), i.e. flipped image (along center vertical axis) + also used the 3 different cameras (left, center and right) with adjustments for the angle.

After the collection process, I had 10485 lines in driving_log.csv, i.e. number of data points = 62430 (6*10485). Preprocessing used to flip image, convert images to numpy arrays and also (as part of Keras model) to scale values. Also did cropping of the image as part of the model. I finally randomly shuffled the data set and put 20 of the data into a validation set, see generator for details. Examples of images (before cropping inside model) is shown below:

Example of center camera image

center Image

Example of flipped center camera image

flippedcenter Image

Example of left camera image

left Image

Example of right camera image

right Image

generator

I used this training data for training the model. The validation helped determine if the model was over or under fitting. The ideal number of epochs was 5 as evidenced by the quick flattening of loss and validation loss (to around 0.03), in earlier runs validation loss increased above training loss when having more epochs. I used an adam optimizer so that manually training the learning rate wasn’t necessary.

3. Challenges

Challenges along the way – found it to be a very hard task, since the model loss and validation loss weren’t good predictors for actual driving performance, also had cases when adding more training data with nice driving data (at the center and far from the edges) actually gave worse results and made the car drive off the road. Other challenges were Keras 2 related, the semantics of parameters in Keras 1 and Keras 2 fooled me a bit using Keras 2, ref the steps_per_epoch. Also had issues with the progress bar not working in Keras 2 in Jupyter notebook, so had to use 3rd party library https://pypi.python.org/pypi/keras-tqdm/2.0.1

Continue Reading

Deep Learning in Energy Production

This blog post has recent publications about use of Deep Learning in Energy Production context (wind, gas and oil), e.g. wind power prediction, turbine risk assessment, reservoir discovery and price forecasting.

Best regards,

Amund Tveit

Wind

Year  Title Author
2017 Short-term Wind Energy Prediction Algorithm Based on SAGA-DBNs  W Fei, WU Zhong
2017 Wind Power Prediction using Deep Neural Network based Meta Regression and Transfer Learning  AS Qureshi, A Khan, A Zameer, A Usman
2017 Wind Turbine Failure Risk Assessment Model Based on DBN  C Fei, F Zhongguang
2017 The optimization of wind power interval forecast  X Yu, H Zang
2016 Deep Learning for Wind Speed Forecasting in Northeastern Region of Brazil  AT Sergio, TB Ludermir
2016 A very short term wind power prediction approach based on Multilayer Restricted Boltzmann Machine  X Peng, L Xiong, J Wen, Y Xu, W Fan, S Feng, B Wang
2016 Short-term prediction of wind power based on deep Long Short-Term Memory  Q Xiaoyun, K Xiaoning, Z Chao, J Shuai, M Xiuda
2016 Deep belief network based deterministic and probabilistic wind speed forecasting approach  HZ Wang, GB Wang, GQ Li, JC Peng, YT Liu
2016 A hybrid wind power prediction method  Y Tao, H Chen
2016 Deep learning based ensemble approach for probabilistic wind power forecasting  H Wang, G Li, G Wang, J Peng, H Jiang, Y Liu
2016 A hybrid wind power forecasting model based on data mining and wavelets analysis  R Azimi, M Ghofrani, M Ghayekhloo
2016 ELM Based Representational Learning for Fault Diagnosis of Wind Turbine Equipment  Z Yang, X Wang, PK Wong, J Zhong
2015 Deep Neural Networks for Wind Energy Prediction  D Díaz, A Torres, JR Dorronsoro
2015 Predictive Deep Boltzmann Machine for Multiperiod Wind Speed Forecasting  CY Zhang, CLP Chen, M Gan, L Chen
2015 Resilient Propagation for Multivariate Wind Power Prediction  J Stubbemann, NA Treiber, O Kramer
2015 Transfer learning for short-term wind speed prediction with deep neural networks  Q Hu, R Zhang, Y Zhou
2014 Wind Power Prediction and Pattern Feature Based on Deep Learning Method  Y Tao, H Chen, C Qiu

Gas

Year  Title Author
2017   Sample Document–Inversion Of The Permeability Of A Tight Gas Reservoir With The Combination Of A Deep Boltzmann Kernel …  L Zhu, C Zhang, Y Wei, X Zhou, Y Huang, C Zhang
2017   Deep Learning: Chance and Challenge for Deep Gas Reservoir Identification  C Junxing, W Shikai
2016   Finite-sensor fault-diagnosis simulation study of gas turbine engine using information entropy and deep belief networks  D Feng, M Xiao, Y Liu, H Song, Z Yang, Z Hu
2015   On Accurate and Reliable Anomaly Detection for Gas Turbine Combustors: A Deep Learning Approach  W Yan, L Yu
2015   A Review of Datasets and Load Forecasting Techniques for Smart Natural Gas and Water Grids: Analysis and Experiments.  M Fagiani, S Squartini, L Gabrielli, S Spinsante
2015   Short-term load forecasting for smart water and gas grids: A comparative evaluation  M Fagiani, S Squartini, R Bonfigli, F Piazza
2015   The early-warning model of equipment chain in gas pipeline based on DNN-HMM  J Qiu, W Liang, X Yu, M Zhang, L Zhang

Oil

Year  Title Author
2017   Development of a New Correlation for Bubble Point Pressure in Oil Reservoirs Using Artificial Intelligent Technique  S Elkatatny, M Mahmoud
2017   A deep learning ensemble approach for crude oil price forecasting  Y Zhao, J Li, L Yu
2016   Automatic Detection and Classification of Oil Tanks in Optical Satellite Images Based on Convolutional Neural Network  Q Wang, J Zhang, X Hu, Y Wang
2015   A Hierarchical Oil Tank Detector With Deep Surrounding Features for High-Resolution Optical Satellite Imagery  L Zhang, Z Shi, J Wu
Continue Reading

Deep Learning for Protein(omics)

This blog post has recent publications related to Deep Learning for proteinomics (the study of proteins). Proteins are a set of molecules in the human (and animal) bodies (probably best known for their role related to muscle mass and in DNA replication).

Wikipedia describes proteins as:

    Proteins (/ˈproʊˌtiːnz/ or /ˈproʊti.ᵻnz/) are large biomolecules, or macromolecules, consisting of one or more long chains of amino acid residues. Proteins perform a vast array of functions within organisms, including catalysing metabolic reactions, DNA replication, responding to stimuli, and transporting molecules from one location to another. Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes, and which usually results in protein folding into a specific three-dimensional structure that determines its activity.

Best regards,
Amund Tveit (WeChat ID: AmundTveit)

Year  Title Author
2017   Deep Recurrent Neural Network for Protein Function Prediction from Sequence  XL Liu
2017   Sequence-based prediction of protein protein interaction using a deep-learning algorithm  T Sun, B Zhou, L Lai, J Pei
2017   Protein Model Quality Assessment: A Machine Learning Approach  K Uziela
2017   Deep convolutional neural networks for detecting secondary structures in protein density maps from cryo-electron microscopy  R Li, D Si, T Zeng, S Ji, J He
2017   Towards recognition of protein function based on its structure using deep convolutional networks  A Tavanaei, AS Maida, A Kaniymattam
2017   Improved protein model quality prediction by changing the target function  K Uziela, D Menendez Hurtado, N Shu, B Wallner
2017   A Novel Model Based On Fcm-Lm Algorithm For Prediction Of Protein Folding Rate  L Liu, M Ma, J Cui
2017   EPSILON-CP: using deep learning to combine information from multiple sources for protein contact prediction  K Stahl, M Schneider, O Brock
2017   Prediction of protein function using a deep convolutional neural network ensemble  EI Zacharaki
2017   Protein Function Prediction using Deep Restricted Boltzmann Machines  X Zou, G Wang, G Yu
2017   Next-Step Conditioned Deep Convolutional Neural Networks Improve Protein Secondary Structure Prediction  A Busia, N Jaitly
2017   Predicting membrane protein contacts from non-membrane proteins by deep transfer learning  Z Li, S Wang, Y Yu, J Xu
2017   A Template-Based Protein Structure Reconstruction Method Using Deep Autoencoder Learning  H Li, Q Lyu, J Cheng
2017   DNpro: A Deep Learning Network Approach to Predicting Protein Stability Changes Induced by Single-Site Mutations  X Zhou, J Cheng
2017   Computational Methods for the Prediction of Drug-Target Interactions from Drug Fingerprints and Protein Sequences by Stacked Auto-Encoder Deep Neural Network  L Wang, ZH You, X Chen, SX Xia, F Liu, X Yan, Y Zhou
2017   Multi-task Deep Neural Networks in Automated Protein Function Prediction  AS Rifaioglu, T Doğan, MJ Martin, R Cetin
2016   AUC-Maximized Deep Convolutional Neural Fields for Protein Sequence Labeling  S Wang, S Sun, J Xu
2016   Evaluation of Protein Structural Models Using Random Forests  R Cao, T Jo, J Cheng
2016   A Protein Domain and Family Based Approach to Rare Variant Association Analysis  TG Richardson, HA Shihab, MA Rivas, MI McCarthy
2016   Protein Sequencing And Neural Network Classification Methods  V Indarni, SK Terala, PV Bhushan, MR Ireddy
2016   Accurate prediction of docked protein structure similarity using neural networks and restricted Boltzmann machines  R Farhoodi, B Akbal
2016   Identification of thermostabilizing mutations for a membrane protein whose three‐dimensional structure is unknown  Y Kajiwara, S Yasuda, Y Takamuku, T Murata
2016   Identification of Genetic Sequences Recognized by Human SC35 Protein Using Artificial Neural Networks: A Deep Learning Approach  AJ Luke, S Fergione
2016   MUST-CNN: A MUltilayer Shift-and-sTitch Deep Convolutional Architecture for Sequence-based Protein Structure Prediction  Z Lin, Y Qi
2016   Protein Secondary Structure Prediction Using Deep Multi-scale Convolutional Neural Networks and Next-Step Conditioning  A Busia, J Collins, N Jaitly
2016   A computational framework for disease grading using protein signatures  E Zerhouni, B Prisacari, Q Zhong, P Wild, M Gabrani
2016   ProtPOS: a python package for the prediction of protein preferred orientation on a surface  JCF Ngai, PI Mak, SWI Siu
2016   DeepQA: Improving the estimation of single protein model quality with deep belief networks  R Cao, D Bhattacharya, J Hou, J Cheng
2016   Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images  V Golkov, MJ Skwark, A Golkov, A Dosovitskiy, T Brox
2016   Protein Secondary Structure Prediction by using Deep Learning Method  Y Wang, H Mao, Z Yi
2016   On the importance of composite protein multiple ligand interactions in protein pockets  S Tonddast‐Navaei, B Srinivasan, J Skolnick
2016   Protein function in precision medicine: deep understanding with machine learning  B Rost, P Radivojac, Y Bromberg
2016   Protein Residue-Residue Contact Prediction Using Stacked Denoising Autoencoders  IV Luttrell, J Bailey
2016   Protein Residue Contacts and Prediction Methods  B Adhikari, J Cheng
2016   RaptorX-Property: a web server for protein structure property prediction.  S Wang, W Li, S Liu, J Xu
2016   AUCpreD: proteome-level protein disorder prediction by AUC-maximized deep convolutional neural fields  S Wang, J Ma, J Xu
2016   Benchmarking Deep Networks for Predicting Residue-Specific Quality of Individual Protein Models in CASP11  T Liu, Y Wang, J Eickholt, Z Wang
2015   Theory, Methods, and Applications of Coevolution in Protein Contact Prediction  J Ma, S Wang
2015   A topological approach for protein classification  Z Cang, L Mu, K Wu, K Opron, K Xia, GW Wei
2015   Application of Learning to Rank to protein remote homology detection  B Liu, J Chen, X Wang
2015   Improving Protein Fold Recognition by Deep Learning Networks  T Jo, J Hou, J Eickholt, J Cheng
2015   Proteins, physics and probability kinematics: a Bayesian formulation of the protein folding problem  T Hamelryck, W Boomsma, J Ferkinghoff
2015   DeepCNF-D: Predicting Protein Order/Disorder Regions by Weighted Deep Convolutional Neural Fields  S Wang, S Weng, J Ma, Q Tang
2015   A deep learning framework for modeling structural features of RNA-binding protein targets  S Zhang, J Zhou, H Hu, H Gong, L Chen, C Cheng
2015   A serum protein test for improved prognostic stratification of patients with myelodysplastic syndrome (MDS)  J Roder, J Löffler
2015   An Overview of Practical Applications of Protein Disorder Prediction and Drive for Faster, More Accurate Predictions  X Deng, J Gumm, S Karki, J Eickholt, J Cheng
2015   A panel of mass spectrometry based serum protein tests for predicting graft-versus-host disease (GvHD) and its severity  H Roder, AC Hoffmann, J Roder, M Koldehoff
2015   Learning Deep Architectures for Protein Structure Prediction  K Baek
2015   Protein secondary structure prediction using deep convolutional neural fields  S Wang, J Peng, J Ma, J Xu
2015   Protein sequence labelling by AUC-maximized Deep Convolutional Neural Fields  S Wang, J Ma, S Sun, J Xu
2015   Fast loop modeling for protein structures  J Zhang, S Nguyen, Y Shang, D Xu, I Kosztin
2015   Introducing Students to Protein Analysis Techniques: Separation and Comparative Analysis of Gluten Proteins in Various Wheat Strains  AL Pirinelli, JC Trinidad, NLB Pohl
2014   Predicting backbone Cα angles and dihedrals from protein sequences by stacked sparse auto‐encoder deep neural network  J Lyons, A Dehzangi, R Heffernan, A Sharma
2014   Improved contact predictions using the recognition of protein like contact patterns.  MJ Skwark, D Raimondi, M Michel, A Elofsson
Continue Reading

Deep Learning for Embedded Systems

This blog post has recent publications related to Deep Learning for Embedded Systems (e.g. computer systems in toys, biometrics, cars, kitchen equipment, medical equipment such as bionic eyes, etc).

Wikipedia defines Embedded systems as:

    An embedded system is a computer system with a dedicated function within a larger mechanical or electrical system, often with real-time computing constraints.[1][2] It is embedded as part of a complete device often including hardware and mechanical parts. Embedded systems control many devices in common use today.[3] Ninety-eight percent of all microprocessors are manufactured as components of embedded systems

Best regards,
Amund Tveit (WeChat ID: AmundTveit – Twitter: atveit)

Year  Title Author
2017   Six Degree-of-Freedom Localization of Endoscopic Capsule Robots using Recurrent Neural Networks embedded into a Convolutional Neural Network  M Turan, A Abdullah, R Jamiruddin, H Araujo
2017   Two-Bit Networks for Deep Learning on Resource-Constrained Embedded Devices  W Meng, Z Gu, M Zhang, Z Wu
2017   14.1 A 2.9 TOPS/W deep convolutional neural network SoC in FD-SOI 28nm for intelligent embedded systems  G Desoli, N Chawla, T Boesch, S Singh, E Guidetti
2017   Characterization of Symbolic Rules Embedded in Deep DIMLP Networks: A Challenge to Transparency of Deep Learning  G Bologna, Y Hayashi
2017   Moving Object Detection in Heterogeneous Conditions in Embedded Systems  A Garbo, S Quer
2016   Re-architecting the on-chip memory sub-system of machine-learning accelerator for embedded devices  Y Wang, H Li, X Li
2016   DELAROSE: A Case Example of the Value of Embedded Course Content and Assessment in the Workplace  JSG Wells, M Bergin, C Ryan
2016   Neurosurgery Conference Experience Embedded within PCOM’s Clinical and Basic Neuroscience Curriculum: An Active Learning Model  J Okun, S Yocom, M McGuiness, M Bell, D Appelt
2016   Scene Parsing using Inference Embedded Deep Networks  S Bu, P Han, Z Liu, J Han
2016   Improving Deep Learning Accuracy with Noisy Autoencoders Embedded Perturbative Layers  L Xia, X Zhang, B Li
2016   Noise Robust Keyword Spotting Using Deep Neural Networks For Embedded Platforms  R Abdelmoula
2016   14.1 A 126.1 mW real-time natural UI/UX processor with embedded deep-learning core for low-power smart glasses  S Park, S Choi, J Lee, M Kim, J Park, HJ Yoo
2016   A wearable mobility aid for the visually impaired based on embedded 3D vision and deep learning  M Poggi, S Mattoccia
2016   Optimizing convolutional neural networks on embedded platforms with OpenCL  A Lokhmotov, G Fursin
2016   Demonstration Abstract: Accelerating Embedded Deep Learning Using DeepX  ND Lane, S Bhattacharya, P Georgiev, C Forlivesi
2016   Feedback recurrent neural network-based embedded vector and its application in topic model  L Li, S Gan, X Yin
2016   Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning  K Wang, S Zhai, H Cheng, X Liang, L Lin
2015   Memory Heat Map: Anomaly Detection in Real-Time Embedded Systems Using Memory Behavior  MK Yoon, S Mohan, J Choi, L Sha
2015   Accelerating real-time embedded scene labeling with convolutional networks  L Cavigelli, M Magno, L Benini
2015   Business meeting training on its head: inverted and embedded learning  E Van Praet
2015   CNN optimizations for embedded systems and FFT  A Vasilyev
2015   Learning Socially Embedded Visual Representation from Scratch  S Liu, P Cui, W Zhu, S Yang
2015   Inter-Tile Reuse Optimization Applied to Bandwidth Constrained Embedded Accelerators  M Peemen, B Mesman, H Corporaal
2015   Emotion recognition from embedded bodily expressions and speech during dyadic interactions  PM Müller, S Amin, P Verma, M Andriluka, A Bulling
2015   Incremental extreme learning machine based on deep feature embedded  J Zhang, S Ding, N Zhang, Z Shi
2015   Utilizing deep neural nets for an embedded ECG-based biometric authentication system  A Page, A Kulkarni, T Mohsenin
2015   A scalable and adaptable probabilistic model embedded in an electronic nose for intelligent sensor fusion  CT Tang, CM Huang, KT Tang, H Chen
Continue Reading

Deep Learning for Magnetic Resonance Imaging (MRI)

Magnetic Resonance Imaging (MRI) can be used in many types of diagnosis e.g. cancer, alzheimer, cardiac and muscle/skeleton issues. This blog post has recent publications of Deep Learning applied to MRI (health-related) data, e.g. for segmentation, detection, demonising and classification.

MRI is described in Wikipedia as:

    Magnetic resonance imaging (MRI) is a medical imaging technique used in radiology to form pictures of the anatomy and the physiological processes of the body in both health and disease. MRI scanners use strong magnetic fields, radio waves, and field gradients to generate images of the organs in the body.

Best regards,
Amund Tveit

Year  Title Author
2017   Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification  S Korolev, A Safiullin, M Belyaev, Y Dodonova
2017   Automatic segmentation of the right ventricle from cardiac MRI using a learning‐based approach  MR Avendi, A Kheradvar, H Jafarkhani
2017   Learning a Variational Network for Reconstruction of Accelerated MRI Data  K Hammernik, T Klatzer, E Kobler, MP Recht
2017   A 2D/3D Convolutional Neural Network for Brain White Matter Lesion Detection in Multimodal MRI  L Roa
2017   On hierarchical brain tumor segmentation in MRI using fully convolutional neural networks: A preliminary study  S Pereira, A Oliveira, V Alves, CA Silva
2017   Classification of breast MRI lesions using small-size training sets: comparison of deep learning approaches  G Amit, R Ben
2017   A deep learning network for right ventricle segmentation in short-axis MRI  GN Luo, R An, KQ Wang, SY Dong, HG Zhang
2017   A novel left ventricular volumes prediction method based on deep learning network in cardiac MRI  GN Luo, GX Sun, KQ Wang, SY Dong, HG Zhang
2017   Classification of MRI data using Deep Learning and Gaussian Process-based Model Selection  H Bertrand, M Perrot, R Ardon, I Bloch
2017   Using Deep Learning to Segment Breast and Fibroglanduar Tissue in MRI Volumes  MU Dalmş, G Litjens, K Holland, A Setio, R Mann
2017   Automatic Liver and Tumor Segmentation of CT and MRI Volumes using Cascaded Fully Convolutional Neural Networks  PF Christ, F Ettlinger, F Grün, MEA Elshaera, J Lipkova
2017   Automated Segmentation of Hyperintense Regions in FLAIR MRI Using Deep Learning  P Korfiatis, TL Kline, BJ Erickson
2017   Automatic segmentation of left ventricle in cardiac cine MRI images based on deep learning  T Zhou, I Icke, B Dogdas, S Parimal, S Sampath
2017   Deep artifact learning for compressed sensing and parallel MRI  D Lee, J Yoo, JC Ye
2017   Deep Generative Adversarial Networks for Compressed Sensing Automates MRI  M Mardani, E Gong, JY Cheng, S Vasanawala
2017   3D Motion Modeling and Reconstruction of Left Ventricle Wall in Cardiac MRI  D Yang, P Wu, C Tan, KM Pohl, L Axel, D Metaxas
2017   Estimation of the volume of the left ventricle from MRI images using deep neural networks  F Liao, X Chen, X Hu, S Song
2017   A fully automatic deep learning method for atrial scarring segmentation from late gadolinium-enhanced MRI images  G Yang, X Zhuang, H Khan, S Haldar, E Nyktari, X Ye
2017   Age estimation from brain MRI images using deep learning  TW Huang, HT Chen, R Fujimoto, K Ito, K Wu, K Sato
2017   Segmenting Atrial Fibrosis from Late Gadolinium-Enhanced Cardiac MRI by Deep-Learned Features with Stacked Sparse Auto-Encoders  S Haldar, E Nyktari, X Ye, G Slabaugh, T Wong
2017   Deep Residual Learning For Compressed Sensing Mri  D Lee, J Yoo, JC Ye
2017   Prostate cancer diagnosis using deep learning with 3D multiparametric MRI  S Liu, H Zheng, Y Feng, W Li
2017   Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions  Z Akkus, A Galimzianova, A Hoogi, DL Rubin
2016   Classification of Alzheimer’s Disease Structural MRI Data by Deep Learning Convolutional Neural Networks  S Sarraf, G Tofighi
2016   De-noising of Contrast-Enhanced MRI Sequences by an Ensemble of Expert Deep Neural Networks  A Benou, R Veksler, A Friedman, TR Raviv
2016   A Combined Deep-Learning and Deformable-Model Approach to Fully Automatic Segmentation of the Left Ventricle in Cardiac MRI  MR Avendi, A Kheradvar, H Jafarkhani
2016   Applying machine learning to automated segmentation of head and neck tumour volumes and organs at risk on radiotherapy planning CT and MRI scans  C Chu, J De Fauw, N Tomasev, BR Paredes, C Hughes
2016   A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI  PV Tran
2016   An Overview of Techniques for Cardiac Left Ventricle Segmentation on Short-Axis MRI  A Krasnobaev, A Sozykin
2016   Stacking denoising auto-encoders in a deep network to segment the brainstem on MRI in brain cancer patients: a clinical study  J Dolz, N Betrouni, M Quidet, D Kharroubi, HA Leroy
2016   Hough-CNN: Deep Learning for Segmentation of Deep Brain Regions in MRI and Ultrasound  F Milletari, SA Ahmadi, C Kroll, A Plate, V Rozanski
2016   Mental Disease Feature Extraction with MRI by 3D Convolutional Neural Network with Multi-channel Input  L Cao, Z Liu, X He, Y Cao, K Li
2016   Deep learning trends for focal brain pathology segmentation in MRI  M Havaei, N Guizard, H Larochelle, PM Jodoin
2016   Identification of Water and Fat Images in Dixon MRI Using Aggregated Patch-Based Convolutional Neural Networks  L Zhao, Y Zhan, D Nickel, M Fenchel, B Kiefer, XS Zhou
2016   Deep MRI brain extraction: A 3D convolutional neural network for skull stripping  J Kleesiek, G Urban, A Hubert, D Schwarz
2016   Active appearance model and deep learning for more accurate prostate segmentation on MRI  R Cheng, HR Roth, L Lu, S Wang, B Turkbey
2016   Recurrent Fully Convolutional Neural Networks for Multi-slice MRI Cardiac Segmentation  RPK Poudel, P Lamata, G Montana
2016   Deep learning predictions of survival based on MRI in amyotrophic lateral sclerosis  HK van der Burgh, R Schmidt, HJ Westeneng
2016   Semantic-Based Brain MRI Image Segmentation Using Convolutional Neural Network  Y Chou, DJ Lee, D Zhang
2016   Abstract WP41: Predicting Acute Ischemic Stroke Tissue Fate Using Deep Learning on Source Perfusion MRI  KC Ho, S El
2016   A new ASM framework for left ventricle segmentation exploring slice variability in cardiac MRI volumes  C Santiago, JC Nascimento, JS Marques
2015   Crohn’s disease segmentation from mri using learned image priors  D Mahapatra, P Schüffler, F Vos, JM Buhmann
2015   Discovery Radiomics for Multi-Parametric MRI Prostate Cancer Detection  AG Chung, MJ Shafiee, D Kumar, F Khalvati
2015   Real-time Dynamic MRI Reconstruction using Stacked Denoising Autoencoder  A Majumdar
2015   q-Space Deep Learning for Twelve-Fold Shorter and Model-Free Diffusion MRI Scans  V Golkov, A Dosovitskiy, P Sämann, JI Sperl
Continue Reading

Deep Learning for Image Super-Resolution (Scale Up)

Scaling down images is a craft, scaling up images is an art

Since in the scaling down to a lower resolution you typically need to remove pixels, but in the case of scaling up you need to invent new pixels. But some Deep Learning models with Convolutional Neural Networks (and frequently Deconvolutional layers) has shown successful to scale up images, this is called Image Super-Resolution. These models are typically trained by taking high resolution images and reducing them to lower resolution and then train in the opposite way. Partially related: Recommend also checking out Odeon et. al’s Distill.pub’s publication: Deconvolution and Checkerboard Artifacts that goes into more detail about the one the core operators used in Image Super-Resolution.

Blog post Illustration Source: Eric Esteve’s 2013 article: Super Resolution bring high end camera image quality to smartphone.

Best regards,

Amund Tveit

Year  Title Author
2017   GUN: Gradual Upsampling Network for single image super-resolution  Y Zhao, R Wang, W Dong, W Jia, J Yang, X Liu, W Gao
2017   Dual Recovery Network with Online Compensation for Image Super-Resolution  S Xia, W Yang, T Zhao, J Liu
2017   A New Single Image Super-resolution Method Based on the Infinite Mixture Model  P Cheng, Y Qiu, X Wang, K Zhao
2017   Underwater Image Super-resolution by Descattering and Fusion  H Lu, Y Li, S Nakashima, H Kim, S Serikawa
2017   Single Image Super-Resolution with a Parameter Economic Residual-Like Convolutional Neural Network  Z Yang, K Zhang, Y Liang, J Wang
2017   Single Image Super-Resolution via Adaptive Transform-Based Nonlocal Self-Similarity Modeling and Learning-Based Gradient Regularization  H Chen, X He, L Qing, Q Teng
2017   Ensemble Based Deep Networks for Image Super-Resolution  Z Huang, L Wang, Y Gong, C Pan
2017   Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network  X Jia, X Xu, B Cai, K Guo
2017   Hyperspectral image super-resolution using deep convolutional neural network  Y Li, J Hu, X Zhao, W Xie, JJ Li
2016   Research on the Natural Image Super-Resolution Reconstruction Algorithm based on Compressive Perception Theory and Deep Learning Model  G Duan, W Hu, J Wang
2016   Image super-resolution with multi-channel convolutional neural networks  Y Kato, S Ohtani, N Kuroki, T Hirose, M Numa
2016   Image super-resolution reconstruction via RBM-based joint dictionary learning and sparse representation  Z Zhang, A Liu, Q Lei
2016   End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks  Y Wang, L Wang, H Wang, P Li
2016   Single image super-resolution using regularization of non-local steering kernel regression  K Zhang, X Gao, J Li, H Xia
2016   Single image super-resolution via blind blurring estimation and anchored space mapping  X Zhao, Y Wu, J Tian, H Zhang
2016   A Versatile Sparse Representation Based Post-Processing Method for Improving Image Super-Resolution  J Yang, J Guo, H Chao
2016   Robust Single Image Super-Resolution via Deep Networks with Sparse Prior.  D Liu, Z Wang, B Wen, J Yang, W Han, T Huang
2016   EnhanceNet: Single Image Super-Resolution through Automated Texture Synthesis  MSM Sajjadi, B Schölkopf, M Hirsch
2016   Is Image Super-resolution Helpful for Other Vision Tasks?  D Dai, Y Wang, Y Chen, L Van Gool
2016   Cluster-Based Image Super-resolution via Jointly Low-rank and Sparse Representation  N Han, Z Song, Y Li
2016   Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network  C Ledig, L Theis, F Huszar, J Caballero, A Aitken
2016   Image super-resolution using non-local Gaussian process regression  H Wang, X Gao, K Zhang, J Li
2016   A hybrid wavelet convolution network with sparse-coding for image super-resolution  X Gao, H Xiong
2016   Amortised MAP Inference for Image Super-resolution  CK Sønderby, J Caballero, L Theis, W Shi, F Huszár
2016   X-Ray fluorescence image super-resolution using dictionary learning  Q Dai, E Pouyet, O Cossairt, M Walton, F Casadio
2016   Image super-resolution based on convolution neural networks using multi-channel input  GY Youm, SH Bae, M Kim
2016   Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution  W Yang, J Feng, J Yang, F Zhao, J Liu, Z Guo, S Yan
2016   Image Super-Resolution by PSOSEN of Local Receptive Fields Based Extreme Learning Machine  Y Song, B He, Y Shen, R Nian, T Yan
2016   Incorporating Image Priors with Deep Convolutional Neural Networks for Image Super-Resolution  Y Liang, J Wang, S Zhou, Y Gong, N Zheng
2015   Single Image Super-Resolution Via Bm3D Sparse Coding  K Egiazarian, V Katkovnik
2015   Learning a Deep Convolutional Network for Light-Field Image Super-Resolution  Y Yoon, HG Jeon, D Yoo, JY Lee, I Kweon
2015   Single Image Super-Resolution via Image Smoothing  Z Liu, Q Huang, J Li, Q Wang
2015   Deeply Improved Sparse Coding for Image Super-Resolution  Z Wang, D Liu, J Yang, W Han, T Huang
2015   Conditioned Regression Models for Non-Blind Single Image Super-Resolution  GRSSM Rüther, H Bischof
2015   How Useful Is Image Super-resolution to Other Vision Tasks?  D Dai, Y Wang, Y Chen, L Van Gool
2015   Learning Hierarchical Decision Trees for Single Image Super-Resolution  JJ Huang, WC Siu
2015   Single image super-resolution by approximated Heaviside functions  LJ Deng, W Guo, TZ Huang
2015   Jointly Optimized Regressors for Image Super-resolution  D Dai, R Timofte, L Van Gool
2015   Single Image Super-Resolution via Internal Gradient Similarity  Y Xian, Y Tian
2015   Image Super-Resolution Using Deep Convolutional Networks  C Dong, CC Loy, K He, X Tang
2015   Coupled Deep Autoencoder for Single Image Super-Resolution  K Zeng, J Yu, R Wang, C Li, D Tao
2015   Single Image Super-Resolution Using Maximizing Self-Similarity Prior  J Li, Y Wu, X Luo
2015   Accurate Image Super-Resolution Using Very Deep Convolutional Networks  J Kim, JK Lee, KM Lee
2015   Deeply-Recursive Convolutional Network for Image Super-Resolution  J Kim, JK Lee, KM Lee
2015   Single Face Image Super-Resolution via Solo Dictionary Learning  F Juefei
2014   Single image super-resolution via L0 image smoothing  Z Liu, Q Huang, J Li, Q Wang
Continue Reading
1 2 3 5