frozenleaves/pytorch
1
0
Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2025-10-20 04:44:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
Page: Tensor and Operator Basics
Pages
A quick guide on how to add and cache dependencies on PyTorch CI Adding and maintaining type annotations in PyTorch Autograd Basics Autograd Onboarding Lab Best Practices to Edit and Compile PyTorch Source Code On Windows Bot commands Boxing and Unboxing in the PyTorch Operator Library Build PyTorch and LibTorch on Windows ARM64 CUDA basics Code review values Codegen and Structured Kernels Continuous Integration Core Frontend Onboarding Cpp API Quick Walkthrough Data Basics DataPipes Testing Requirements Debugging CI Failures without SSH Access Debugging Windows with Remote Desktop or CDB (CLI windbg) on CircleCI Debugging using with ssh for Github Actions Dev Infra Office Hours Developer FAQ Dispatcher Structured Kernels Lab Docker image build on CircleCI Docstring Guidelines Getting help as a contributor Guide for adding type annotations to PyTorch Home How to propose feature changes to PyTorch How to support `torch.set_deterministic()` in PyTorch operators How to use TensorIterator Life of a Tensor MPS Backend Memory format propagation rules Modular components for benchmarking PyTorch snippets. (Experimental) Module Onboarding Lab Multiprocessing Technical Notes OpInfos FAQ Operators with Channels Last support Public API definition and documentation Pull request review etiquette PyTorch's Python Frontend Backward and Forward Compatibility Policy PyTorch AutoLabel Bot PyTorch Basics PyTorch CI Metrics Dashboards: the HUD PyTorch Data Flow and Interface Diagram PyTorch IR PyTorch ONNX Exporter Code Reviews and Duty Rotation PyTorch ONNX Topics PyTorch ONNX exporter PyTorch OSS benchmark infra PyTorch Ops to oneDNN Functions Mapping PyTorch Versions PyTorch Workflow Cheatsheet PyTorch dispatcher walkthrough Pytorch Training Loops Running and writing tests Sharing design documents for discussion Software Architecture for c10 TH to ATen porting guide Tensor and Operator Basics The PyTorch Contribution Process The Ultimate Guide to PyTorch Contributions The torch.fft module in PyTorch 1.7 Troubleshooting Documentation Build Troubleshooting Using hud.pytorch.org What is considered a SEV? Where or how should I add documentation Writing Python in cpp (a manifesto) Writing memory format aware operators Writing tests in PyTorch 1.8 clang format function transforms (aka torch.func, functorch) lintrunner nn Basics torch.nn Module Documentation Style Guide torch.onnx Namespacing vmap Basics vmap Onboarding Lab
Clone
6
Tensor and Operator Basics
jhelsby edited this page 2024-08-31 14:32:34 +01:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Scope

  • Understand what a tensor is (including stride, dtype, and layout)
  • Understand what an operator is in PyTorch
  • Understand what views are
  • Understand how to author an operator in PyTorch
  • Understand how to test operators in PyTorch
  • Understand what TensorIterator is

What is a tensor?

Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters. Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and NumPy arrays can often share the same underlying memory, eliminating the need to copy data. Tensors are also optimized for automatic differentiation. (Source: PyTorch Basics Tutorial.)

A Tensor consists of:

  • data_ptr, a pointer to a chunk of memory
  • some sizes metadata
  • some strides metadata
  • a storage offset

What is an operator?

A kernel is a function that accepts Tensors and/or raw pointers to memory and performs a useful computation (for example, matrix multiplication, attention, etc).

An operator is glue code for the PyTorch runtime that tells it about the computation. A single operator can be associated with multiple kernels (for example, torch.add has a kernel for CPU and a kernel for CUDA). The glue code is necessary to get PyTorch subsystems (like torch.compile and torch.autograd) to compose with the computation.

Standalone kernels may work directly with PyTorch but will not compose with the majority of PyTorch subsystems. In order to get them to compose, please register an operator for them.

(Source: The Custom Operator Manual, provided in the official PyTorch Custom Operators documentation.)

How to author an operator

Comprehensive guide

TensorIterator

  • Read through the colab notebook (link)
  • Listen to the podcast (link)

Learn about view operations

  • Read through the colab notebook (link)
  • Try out the Tensor Views lab (link)

Bonus Lab

  • After completing the views lab if you have extra time feel free to work through Sasha Rush's Tensor Puzzles

Next

Unit 2: Tensors, Operators, and Testing - Running and writing tests

I would love to contribute to PyTorch!