$$

Personal highlights from Danny Hernandez’s talk at EAG SF 2022.

• If you believed in Moore’s law in the ’70s you could have easily predicted things like smartphones
  • and it works have been relatively sound because the trend has been going on for a while in terms of hardware in general (if you don’t look only at transistors but computers in general)
• Overall progress is the combination of three exponentials:
  • hardware
    • right now, GPU performance is doubling every 2 years (classic Moore’s)
    • might slow down to every 4 years (which is where CPUs are right now)
  • spending
  • algorithms
    • hardest to understand, has spikes
    • the transformer was 50-100x (!!)
    • scaling laws / Chinchilla was 3x
    • comparison point: optimization software for mixed integer programming