Local LLMs with Ollama & llama.cpp Interview Questions Hindi mein

जवाब: Prompts and data never have to leave your machine

Privacy and control are the core advantages. Local models are typically smaller and less capable than frontier hosted models, and hardware planning becomes your job.

जवाब: To keep attention keys and values for earlier tokens so each new token does not reprocess the whole sequence

The KV cache holds per-token attention keys and values, so decode only computes the new token. It grows with context length, which is why long contexts need more memory.

जवाब: Fewer bytes must be read from memory for each generated token

Decode is mostly memory-bandwidth bound. A 4-bit model has the same architecture but roughly a quarter of the bytes to stream per token.

जवाब: The instruct (chat) fine-tuned variant

Instruct models are trained to follow instructions in a chat format. Base models only continue text and often do not answer questions directly.

जवाब: Metadata such as architecture, tokenizer and chat template

GGUF stores tensors plus metadata (architecture, hyperparameters, tokenizer, special tokens, often the chat template), so one file is enough to run the model.

जवाब: A medium mix that keeps some sensitive tensors at higher precision

K-quant variants come in S, M and L mixes. M is a medium mix where certain tensors get more bits, balancing size and quality.

जवाब: The CLI sends HTTP requests to the local Ollama server, which loads and runs the model

Ollama is client-server. The CLI talks to the server on port 11434, which manages model loading and inference.

जवाब: ollama ps

`ollama ps` lists running (loaded) models with memory use and processor split. `ollama list` only shows what is downloaded.

जवाब: An embedding model such as nomic-embed-text

Embedding models map text to vectors for similarity search. Chat and coder models generate text.

जवाब: It reuses the existing weight layers and adds only the new configuration

Ollama stores models as layers. A derived model points to the same weight blob and adds small layers for system prompt, parameters and template. No training happens.

जवाब: A low temperature such as 0 to 0.2

Extraction wants deterministic, most-likely tokens. Low temperature (or greedy decoding) reduces random variation and format errors.

जवाब: A wrong chat template or missing stop token

If the template or stop tokens do not match training, the model never sees its end-of-turn marker and keeps writing the conversation.

जवाब: A stream of newline-delimited JSON objects

Streaming is on by default and arrives as NDJSON chunks; set "stream": false for one JSON response.

जवाब: Create Client(host="http://other-machine:11434") and call its methods

A Client takes a host argument; the default client also honours the OLLAMA_HOST environment variable.

जवाब: Set base_url to http://localhost:11434/v1, use a local model name, and pass any placeholder api_key

Ollama serves an OpenAI-compatible API under /v1. The SDK needs some api_key value, which Ollama ignores.

जवाब: llama-server

llama-server is the HTTP server with /v1 endpoints and a web UI. The others quantize, benchmark and measure quality.

जवाब: Offload up to 99 layers (effectively all) to the GPU

-ngl / --n-gpu-layers controls how many layers live on the GPU. A number larger than the layer count means the whole model is offloaded.

जवाब: About 2048 tokens each

With the classic per-slot split, the total context is divided among parallel slots: 8192 / 4 = 2048 tokens per slot.

जवाब: That the output's structure is valid for the schema, as long as generation completes

Masking invalid tokens guarantees the shape, not the truth of the values. Truncation can still cut the output short.

जवाब: Print the retrieved chunks to see whether the right passage was found

Most RAG failures are retrieval failures. If the correct chunk is not in the prompt, no chat model can answer from it.

जवाब: Your application code, after reading and validating the model's tool call

The model only emits a structured request. Your code decides whether and how to execute it, then returns the result as a tool message.

जवाब: The KV cache, which scales linearly with context length

Weights are fixed by the file. The KV cache stores keys and values for every token of context, so 8 times the context means about 8 times the cache.

जवाब: Layers left in system RAM run at much lower bandwidth and bottleneck every token

Each token must pass through every layer. The CPU-resident layers are slow, so they dominate the time per token.

जवाब: The context window is too small, so older tokens are truncated

When the conversation exceeds num_ctx, the oldest tokens are dropped. Increase the context (memory permitting) or summarize older turns.

जवाब: So downloaded models persist across container restarts instead of being re-downloaded

Models live in that directory. Without a volume they vanish with the container and must be pulled again.

जवाब: Anyone who can reach the port can use the API, because Ollama has no built-in authentication

Binding to all interfaces exposes an unauthenticated API. Use a firewall, SSH tunnel or authenticating reverse proxy.

जवाब: Run both on the same labelled set of real tickets with fixed settings and compare accuracy and speed

A task-specific evaluation with fixed settings answers the question directly; perplexity is a useful secondary signal.

जवाब: Ollama at 127.0.0.1:11434, llama-server at 127.0.0.1:8080

Ollama listens on localhost port 11434 and llama-server on localhost port 8080 by default; both bind to loopback unless configured otherwise.

जवाब: At roughly 40 GB the weights do not fit in 24 GB of VRAM, so many layers run from system RAM

70B x about 4.8 bits / 8 is roughly 40 GB. The overflow lives in system RAM, and those layers bottleneck every token.

जवाब: Each parallel slot needs its own KV cache, so total memory grew and layers were pushed off the GPU

More parallel slots multiply KV-cache memory. With less VRAM left for weights, Ollama offloads fewer layers to the GPU.