</>SkillByAI
🇬🇧 English
Test our app

PyTorch interview questions

33 questions from the PyTorch course, each with the correct answer and an explanation. Try to answer before you open the answer.

Mental model and setup

1. What does it mean that PyTorch runs in eager mode by default?

  1. You must compile the whole model into a static graph before running it
  2. Operations are queued and run only when you call `torch.run()`
  3. Each operation executes immediately as Python reaches it, so you can print and debug normally
  4. Gradients are computed eagerly before the forward pass
Show answer

Answer: Each operation executes immediately as Python reaches it, so you can print and debug normally

In eager mode every line runs as soon as Python executes it. That is why you can use print, debuggers and ordinary control flow inside a model.

Lesson: What PyTorch is and why people use it

2. You install a CUDA-enabled PyTorch wheel with pip on Linux. What must already be on the machine for the GPU to work?

  1. The full CUDA toolkit including nvcc
  2. A matching cuDNN download from the NVIDIA website
  3. Nothing; PyTorch ships its own GPU driver
  4. A compatible NVIDIA driver
Show answer

Answer: A compatible NVIDIA driver

The wheels bundle the CUDA runtime and cuDNN libraries. The one thing they cannot ship is the kernel driver, so a compatible NVIDIA driver must be installed.

Lesson: Installing PyTorch and choosing a device

3. In gradient descent, what does the learning rate control?

  1. The size of each parameter update step
  2. How many examples are in one batch
  3. How many epochs training runs for
  4. The direction in which parameters move
Show answer

Answer: The size of each parameter update step

The update is `w - lr * grad`. The gradient gives the direction; the learning rate scales how far you move.

Lesson: The training loop idea, without PyTorch

Tensor foundations

4. What is the dtype of `torch.tensor([1, 2, 3])`?

  1. torch.float32
  2. torch.int64
  3. torch.int32
  4. torch.float64
Show answer

Answer: torch.int64

Python integers are inferred as `torch.int64`. Python floats would give `torch.float32`.

Lesson: Creating tensors: shape, dtype and device

5. `x` has shape `(32, 10)` (32 examples, 10 features). What is the shape of `x.sum(dim=1)`?

  1. (10,)
  2. (1, 10)
  3. (32,)
  4. (32, 1)
Show answer

Answer: (32,)

`dim=1` (the features) collapses, leaving one value per example: shape `(32,)`. With `keepdim=True` it would be `(32, 1)`.

Lesson: Tensor operations and reductions

6. Which statement about `view` and `reshape` is correct?

  1. `view` always copies the data, `reshape` never does
  2. Both always copy, so changing the result never affects the original
  3. `reshape` only works on contiguous tensors
  4. `view` never copies and errors if it cannot, while `reshape` copies when a view is impossible
Show answer

Answer: `view` never copies and errors if it cannot, while `reshape` copies when a view is impossible

`view` is strict: same storage or an error. `reshape` is flexible: it returns a view when the layout allows and a copy otherwise.

Lesson: Indexing, views and reshaping

7. What is the result shape of adding tensors with shapes `(4, 1, 3)` and `(5, 1)`?

  1. (4, 5, 3)
  2. It raises an error
  3. (4, 1, 3)
  4. (5, 4, 3)
Show answer

Answer: (4, 5, 3)

Align from the right: 3 vs 1 gives 3, 1 vs 5 gives 5, and 4 vs missing gives 4. The result is `(4, 5, 3)`.

Lesson: Broadcasting rules and the silent shape bug

Autograd: automatic gradients

8. Starting from the same leaf `x` (with `requires_grad=True`), you run the forward pass twice, building a new graph each time, and call `backward()` on each result without clearing `.grad` in between. What is in `x.grad`?

  1. Only the gradient from the second call
  2. The sum of the gradients from both calls
  3. None, because the second call resets it
  4. An error is raised on the second `backward()` call
Show answer

Answer: The sum of the gradients from both calls

Backward adds into `.grad`. Each graph is used once, so no error is raised, and with no clearing, `x.grad` holds the sum. That is why training loops call `optimizer.zero_grad()`.

Lesson: requires_grad, backward and .grad

9. By default, which tensors have their `.grad` attribute populated after `loss.backward()`?

  1. Every tensor in the graph, including intermediates
  2. Only the loss tensor itself
  3. Only tensors created inside `torch.no_grad()`
  4. Leaf tensors that have `requires_grad=True`
Show answer

Answer: Leaf tensors that have `requires_grad=True`

Intermediate gradients are discarded to save memory unless you call `retain_grad()`. Leaves with requires_grad, such as model parameters, keep theirs in `.grad`.

Lesson: The dynamic computation graph

10. You want to record the loss value every step for a plot, without keeping the computation graph alive. What should you store?

  1. The `loss` tensor itself, appended to a list
  2. `torch.tensor(loss, requires_grad=True)`
  3. `loss.item()`
  4. The result of calling `loss.backward()`
Show answer

Answer: `loss.item()`

`loss.item()` returns a plain Python float with no link to the graph. Storing the tensor keeps the whole graph and its memory alive.

Lesson: no_grad, inference_mode, detach and zeroing gradients

Building models with torch.nn

11. Why should you call `model(x)` rather than `model.forward(x)`?

  1. `model(x)` runs registered hooks around `forward`, which calling `forward` directly skips
  2. `model.forward(x)` does not compute gradients
  3. `model(x)` automatically moves `x` to the GPU
  4. `model.forward(x)` always runs in evaluation mode
Show answer

Answer: `model(x)` runs registered hooks around `forward`, which calling `forward` directly skips

`nn.Module.__call__` handles hooks and then calls `forward`. Calling `forward` directly bypasses that machinery; gradients and devices are unaffected.

Lesson: nn.Module: the shape of every model

12. Why are non-linear activations placed between Linear layers?

  1. They reduce the number of parameters
  2. Without them, stacked Linear layers are equivalent to a single Linear layer
  3. They are required for autograd to compute gradients
  4. They convert logits into probabilities
Show answer

Answer: Without them, stacked Linear layers are equivalent to a single Linear layer

A composition of linear maps is linear, so depth would add nothing. Activations make the network able to represent non-linear functions.

Lesson: Common layers: Linear, Conv2d, activations and Sequential

13. Your model outputs logits of shape `(N, 5)` and your labels are integers from 0 to 4. Which setup is correct?

  1. Apply softmax, then `nn.MSELoss` against the labels
  2. `nn.BCEWithLogitsLoss` with the integer labels
  3. `nn.CrossEntropyLoss` on the raw logits with `int64` labels
  4. Apply softmax, then `nn.CrossEntropyLoss`
Show answer

Answer: `nn.CrossEntropyLoss` on the raw logits with `int64` labels

CrossEntropyLoss expects raw logits and class-index targets; it applies log_softmax itself. Adding softmax first double-applies it and weakens gradients.

Lesson: Loss functions and the logits rule

Training: optimizers, the loop and evaluation

14. How does AdamW differ from Adam?

  1. AdamW does not use momentum
  2. AdamW uses a fixed learning rate that cannot be scheduled
  3. AdamW only works on GPUs
  4. AdamW applies weight decay directly to the weights instead of adding it to the gradient
Show answer

Answer: AdamW applies weight decay directly to the weights instead of adding it to the gradient

AdamW decouples weight decay from the adaptive gradient update. Momentum-like running averages and scheduling work the same as in Adam.

Lesson: Optimizers and learning-rate schedules

15. Which order of calls inside the batch loop is correct?

  1. `backward()`, `zero_grad()`, forward and loss, `step()`
  2. `zero_grad()`, forward and loss, `backward()`, `step()`
  3. forward and loss, `step()`, `backward()`, `zero_grad()`
  4. `step()`, forward and loss, `zero_grad()`, `backward()`
Show answer

Answer: `zero_grad()`, forward and loss, `backward()`, `step()`

Clear gradients, compute the loss, back-propagate, then update. Any other order either uses stale gradients or wipes the new ones before they are used.

Lesson: The canonical training loop

16. What does calling `model.eval()` change?

  1. The behaviour of layers such as Dropout and BatchNorm
  2. It disables gradient tracking for all operations
  3. It moves the model to the CPU
  4. It freezes the parameters so the optimizer cannot change them
Show answer

Answer: The behaviour of layers such as Dropout and BatchNorm

`eval()` flips the training flag that Dropout and BatchNorm read. Gradient tracking needs `no_grad()` or `inference_mode()` separately.

Lesson: Evaluation: train vs eval mode and metrics

Data pipelines: Dataset, DataLoader and transforms

17. Which two methods must a map-style `Dataset` implement?

  1. `__iter__` and `__next__`
  2. `load` and `transform`
  3. `forward` and `backward`
  4. `__len__` and `__getitem__`
Show answer

Answer: `__len__` and `__getitem__`

Map-style datasets are indexed: `__len__` gives the size and `__getitem__` returns one sample. `__iter__` belongs to `IterableDataset`.

Lesson: Writing a custom Dataset

18. What does `num_workers=4` do in a `DataLoader`?

  1. Trains the model on four GPUs
  2. Loads samples in four background processes while the main process trains
  3. Splits each batch into four smaller batches
  4. Repeats each epoch four times
Show answer

Answer: Loads samples in four background processes while the main process trains

Workers are separate processes that run `__getitem__` and collation in parallel, so the next batches are ready when the training step needs them.

Lesson: DataLoader: batching, shuffling and workers

19. Why does the validation pipeline leave out `RandomHorizontalFlip` and `RandomResizedCrop`?

  1. Validation images are already flipped
  2. Random transforms are not supported outside training
  3. Validation must be deterministic so scores are comparable and reflect real inputs
  4. Random transforms only work with `ImageFolder`
Show answer

Answer: Validation must be deterministic so scores are comparable and reflect real inputs

Augmentation is a training-time regulariser. Validation should measure the model on consistent, unaltered inputs.

Lesson: Image transforms and augmentation

Real-world models

20. What is the main benefit of ending a CNN with `nn.AdaptiveAvgPool2d(1)` before the Linear head?

  1. It removes the need for a loss function
  2. It makes the head independent of the input image size
  3. It doubles the number of channels
  4. It replaces BatchNorm
Show answer

Answer: It makes the head independent of the input image size

Global average pooling always outputs one value per channel, so the Linear layer's input size stays the same whatever the height and width of the input.

Lesson: A convolutional image classifier

21. You have 300 labelled images across 5 classes. What is a sensible first approach?

  1. Train a ResNet from random initialisation for many epochs
  2. Fine-tune every layer with a large learning rate
  3. Use a pretrained model without changing its 1000-class head
  4. Freeze a pretrained backbone and train a new 5-class head
Show answer

Answer: Freeze a pretrained backbone and train a new 5-class head

With little data, feature extraction is fast and resists overfitting. The head must be replaced to output 5 classes.

Lesson: Transfer learning with pretrained models

22. What must the input to `nn.Embedding` be?

  1. Float vectors of size `embed_dim`
  2. An integer tensor of token indices
  3. One-hot encoded float tensors
  4. Normalised values between 0 and 1
Show answer

Answer: An integer tensor of token indices

An embedding is a lookup table indexed by integer ids. Passing floats raises an error; one-hot vectors are unnecessary.

Lesson: Embeddings and sequence models

GPUs and performance

23. A model is on the GPU. You call `x.to("cuda")` without assigning the result, then run `model(x)`. What happens?

  1. It works, because `.to()` moved `x` in place
  2. It works, but runs on the CPU
  3. It fails with a device mismatch, because `x` itself is still on the CPU
  4. PyTorch moves the model back to the CPU automatically
Show answer

Answer: It fails with a device mismatch, because `x` itself is still on the CPU

`tensor.to()` returns a new tensor. Without reassignment, `x` stays on the CPU and the forward pass mixes devices.

Lesson: Devices: moving models and data to the GPU

24. Why is a gradient scaler used with float16 mixed precision?

  1. Small gradient values can underflow to zero in float16, and scaling the loss keeps them representable
  2. float16 cannot represent negative numbers
  3. The optimizer cannot read float16 gradients
  4. Scaling makes the learning rate larger
Show answer

Answer: Small gradient values can underflow to zero in float16, and scaling the loss keeps them representable

float16's limited range flushes tiny values to zero. Scaling the loss scales every gradient up; they are unscaled before the step.

Lesson: Mixed precision with autocast and GradScaler

25. Training is slow and `nvidia-smi` shows the GPU mostly idle. What is the most likely first fix?

  1. Switch from Adam to SGD
  2. Add more layers so the GPU has more work
  3. Call `torch.cuda.synchronize()` after every batch
  4. Speed up data loading, for example with more `num_workers` and `pin_memory=True`
Show answer

Answer: Speed up data loading, for example with more `num_workers` and `pin_memory=True`

An idle GPU is usually waiting for the input pipeline. Parallel loading and pinned memory keep batches ready; extra synchronisation would slow things further.

Lesson: torch.compile and a faster input pipeline

Saving, debugging, security and deployment

26. What is the recommended way to save a trained model for later use?

  1. `torch.save(model)` to pickle the whole object
  2. `torch.save(model.state_dict(), path)`, then rebuild the class and call `load_state_dict`
  3. Print the weights and copy them into a text file
  4. Save only the optimizer's state dict
Show answer

Answer: `torch.save(model.state_dict(), path)`, then rebuild the class and call `load_state_dict`

A state dict holds only tensors keyed by name, so it survives refactoring. Pickling the whole model ties the file to your exact class and module path.

Lesson: Saving and loading with state_dict

27. Your training loss stays flat from the first epoch. Which test best separates a code bug from a data-size problem?

  1. Try to overfit a single small batch to near-zero loss
  2. Double the number of epochs
  3. Switch to a larger pretrained model
  4. Add more data augmentation
Show answer

Answer: Try to overfit a single small batch to near-zero loss

A correct model and loss can memorise one batch. If it cannot, something in the code is broken; the other options assume the code already works.

Lesson: Debugging training: shapes, NaNs and silent failures

28. What does `torch.use_deterministic_algorithms(True)` do when an operation has no deterministic implementation?

  1. Silently uses the nondeterministic version
  2. Falls back to running that operation on the CPU
  3. Raises an error naming the operation
  4. Skips the operation
Show answer

Answer: Raises an error naming the operation

It raises a RuntimeError so you know reproducibility cannot be guaranteed, instead of quietly running nondeterministic code.

Lesson: Reproducibility and randomness

29. Before trusting an exported model in production, what should you do?

  1. Retrain it from scratch in the export format
  2. Compare its outputs with the original eager model on sample inputs
  3. Convert all weights to int64
  4. Delete the original PyTorch weights
Show answer

Answer: Compare its outputs with the original eager model on sample inputs

Export can change behaviour (for example through unsupported operations or mode mistakes). Comparing outputs with `torch.testing.assert_close` catches that early.

Lesson: Exporting and serving models

Revision and interview prep

30. Which pair is the correct way to evaluate a model without wasting memory and with the right layer behaviour?

  1. `model.train()` with `torch.no_grad()`
  2. `model.eval()` alone
  3. `torch.inference_mode()` alone
  4. `model.eval()` together with `torch.inference_mode()` (or `no_grad()`)
Show answer

Answer: `model.eval()` together with `torch.inference_mode()` (or `no_grad()`)

`eval()` fixes Dropout and BatchNorm behaviour; `inference_mode()` or `no_grad()` stops graph building. You need both.

Lesson: PyTorch cheat sheet

31. An interviewer asks why `CrossEntropyLoss` should receive raw logits. Which answer is correct?

  1. It applies log_softmax internally, so an extra softmax distorts the loss and weakens gradients
  2. Logits are always positive, which the loss requires
  3. Softmax outputs cannot be moved to the GPU
  4. Raw logits make the loss independent of the batch size
Show answer

Answer: It applies log_softmax internally, so an extra softmax distorts the loss and weakens gradients

CrossEntropyLoss combines log_softmax and negative log-likelihood. Feeding probabilities applies softmax twice.

Lesson: Common interview and exam questions

32. In the script above, which bug would raise an exception on the very first batch?

  1. The missing `optimizer.zero_grad()`
  2. `shuffle=False` on the training loader
  3. `x.to("cuda")` without reassigning `x`
  4. The softmax before `CrossEntropyLoss`
Show answer

Answer: `x.to("cuda")` without reassigning `x`

`x` stays on the CPU while the model is on the GPU, so the first forward pass fails with a device mismatch. The other bugs run silently and only hurt results.

Lesson: Self-check: spot the bugs

33. You load a NumPy `float64` array with `torch.from_numpy`, pass it to an `nn.Linear` model and get *expected scalar type Float but found Double*. What is the fix?

  1. Convert the tensor with `.float()` before passing it to the model
  2. Call `model.double()` on every model you ever train
  3. Use `torch.no_grad()` around the forward pass
  4. Move the model to the GPU
Show answer

Answer: Convert the tensor with `.float()` before passing it to the model

The data is float64 while the model's weights are float32. Converting the input with `.float()` is the standard fix; switching the whole model to float64 is slower and rarely needed.

Lesson: Mixed self-check

← Back to the course