Every failing sample below is the real model output, with the facts it missed or invented. Trial 0 runs at temperature 0, trials 1 and 2 at 0.7 — so a task passing 1/3 is a sampling-luck pass, not a reliable one. Models are ordered best first. Other technologies: C · Python · Bash · research · how to reproduce.
Setup and model serving are on the models index — including a script-free version using nothing but curl and a compiler; with a server running, this is the exact command that produced the samples below:
# research paper: extract NFS facts from ~2,100 words of RHEL 10 docs
# (distractor paragraphs included), 3 trials, graded by exact fact match
python scripts/eval_research.py --target <model> --trials 3 \
--dump-failures results/failuresServed via llama.cpp CUDA · 28.1 tok/s · 29.8 GB RAM · coding total 111/126
No failures.
Served via llama.cpp CUDA · 51.7 tok/s · 2.0 GB RAM · coding total 108/126
No failures.
Served via llama.cpp CUDA · 27.8 tok/s · 1.0 GB RAM · coding total 108/126
- **NFS with TLS support added (RHEL 10.0)** – TLS is now fully supported for NFS, encrypting RPC traffic between clients and servers via kernel TLS (kTLS). Improves security without requiring application changes; kTLS itself remains a Technology Preview.
- **NFS client supports larger rsize/wsize (up to 1MB) (RHEL 10.0)** – Maximum read/write buffer sizes increased to 1,048,576 bytes on both client and server. Enhances NFS performance by allowing larger data transfers per operation.
- **IPv6 address support fixed in nfs:// URLs for ReaR (RHEL 10.0)** – Square brackets around IPv6 addresses in `nfs://` URLs are no longer misinterpreted as shell metacharacters. Enables correct use of IPv6 NFS URLs in backup/restore configurations.
- **FS-Cache client-side caching documentation updated (RHEL 10.0)** – Clarifies use of `cachefiles` and `cachefilesd` for NFS caching, including supported backends (XFS, ext3, ext4) and behavior with partial caching and I/O error handling. Helps optimize NFS client performance and reduce server load.- **NFS with TLS support added (RHEL 10.0)** – TLS is now fully supported for NFS, encrypting RPC traffic between clients and servers. This improves security without requiring application changes, though it relies on kTLS (provided as a Technology Preview).
- **NFS client supports `vers` as alias for `nfsvers` (RHEL 10.0)** – The `vers` mount option is now equivalent to `nfsvers`, included for compatibility. This simplifies configuration and aligns with standard NFS mount option naming.
- **Increased default and maximum `rsize`/`wsize` values for NFS (RHEL 10.0)** – The maximum transfer size for NFS read/write operations is now 1,048,576 bytes on both client and server. Larger I/O sizes can improve performance by reducing protocol overhead and increasing throughput.
- **IPv6 address support in `nfs://` URLs for ReaR (RHEL 10.0)** – Fixed parsing of IPv6 addresses enclosed in square brackets in `nfs://` URLs used in `BACKUP_URL` and `OUTPUT_URL`. This enables correct backup and restore operations over NFS when using IPv6.
- **FS-Cache integration with NFS remains supported (RHEL 10.0)** – Client-side caching via FS-Cache and `cachefilesd` continues to be available for NFS mounts. This can reduce server load and improve performance for repeated reads, especially over high-latency networks.- **NFS with TLS support added (RHEL 10.0)** – TLS is now fully supported for NFS, encrypting RPC traffic between clients and servers. This improves security without requiring application changes, though it relies on kTLS (provided as a Technology Preview).
- **NFS client supports `vers` as alias for `nfsvers` (RHEL 10.0)** – The `vers` mount option is now equivalent to `nfsvers`, included for compatibility. This simplifies configuration and aligns with standard NFS mount option naming.
- **Increased default and maximum `rsize`/`wsize` values for NFS (RHEL 10.0)** – The maximum transfer size for NFS read/write operations is now 1,048,576 bytes on both client and server. Larger I/O sizes can improve performance by reducing protocol overhead and increasing throughput.
- **IPv6 address support in `nfs://` URLs for ReaR (RHEL 10.0)** – Fixed parsing of IPv6 addresses enclosed in square brackets in `nfs://` URLs used in `BACKUP_URL` and `OUTPUT_URL`. This enables correct backup and restore operations over NFS when using IPv6.
- **FS-Cache integration with NFS remains supported (RHEL 10.0)** – Client-side caching via FS-Cache and `cachefilesd` continues to be available for NFS mounts. This can reduce server load and improve performance for repeated reads, especially over high-latency networks.Served via llama.cpp CUDA · 48.9 tok/s · 1.3 GB RAM · coding total 107/126
No failures.
Served via llama.cpp CUDA · 16.1 tok/s · 3.8 GB RAM · coding total 105/126
- **NFS with TLS support**: RHEL 10 fully supports NFS with Transport Layer Security (TLS) for encrypting RPC traffic between clients and servers, enhancing security for NFS communications. This matters because it protects data in transit from eavesdropping and tampering, though it relies on kernel TLS (kTLS), which remains a Technology Preview for general use.
- **IPv6 address handling in NFS URLs**: Fixed an issue where IPv6 addresses enclosed in square brackets (e.g., `[2001:db8:ca2:6::101]`) in `nfs://` or `sshfs://` URLs were incorrectly interpreted as shell metacharacters, causing ReaR to fail. Now, such URLs are parsed correctly, enabling reliable use of IPv6 in NFS-based backup and output configurations without workarounds.
- **Client-side caching via FS-Cache**: NFS clients can now leverage FS-Cache with the `cachefiles` backend to cache NFS content locally, improving read performance, reducing server and network load, and enabling efficient local data retrieval. This matters because it significantly boosts NFS performance for repetitive read workloads while maintaining transparency and handling partial caching and I/O errors gracefully. Supported cache back ends include XFS, ext3, and ext4 with extended attributes.- **NFS with TLS support**: RHEL 10 fully supports NFS with Transport Layer Security (TLS) for encrypting RPC traffic between clients and servers, enhancing security for NFS communications. This matters because it protects data in transit from eavesdropping and tampering, though it relies on kernel TLS (kTLS), which remains a Technology Preview for general use.
- **IPv6 address handling in NFS URLs**: Fixed an issue where IPv6 addresses enclosed in square brackets (e.g., `[2001:db8:ca2:6::101]`) in `nfs://` or `sshfs://` URLs were incorrectly interpreted as shell metacharacters, causing ReaR to fail. Now, such URLs are parsed correctly, enabling reliable use of IPv6 in NFS-based backup and output configurations without workarounds.
- **Client-side caching via FS-Cache**: NFS clients can now leverage FS-Cache with the `cachefiles` backend to cache NFS content locally, improving read performance, reducing server and network load, and enabling efficient local data retrieval. This matters because it significantly boosts NFS performance for repetitive read workloads while maintaining transparency and handling partial caching and I/O errors gracefully. Supported cache back ends include XFS, ext3, and ext4 with extended attributes.- **NFS with TLS support**: RHEL 10 fully supports NFS with Transport Layer Security (TLS) for encrypting RPC traffic between clients and servers, enhancing security for NFS communications. This matters because it protects data in transit from eavesdropping and tampering, though it relies on kernel TLS (kTLS), which remains a Technology Preview for general use.
- **IPv6 address handling in NFS URLs**: Fixed an issue where IPv6 addresses enclosed in square brackets (e.g., `[2001:db8:ca2:6::101]`) in `nfs://` or `sshfs://` URLs were incorrectly interpreted as shell metacharacters, causing ReaR to fail. Now, such URLs are parsed correctly, enabling reliable use of IPv6 in NFS-based backup and output configurations without workarounds.
- **Client-side caching via FS-Cache**: NFS clients can now leverage FS-Cache with the `cachefiles` backend to cache NFS content locally, improving read performance, reducing server and network load, and enabling efficient local data retrieval. This matters because it significantly boosts NFS performance for repetitive read workloads while maintaining transparency and handling partial caching and I/O errors gracefully. Supported cache back ends include XFS, ext3, and ext4 with extended attributes.Served via llama.cpp CUDA · 81.8 tok/s · 1.3 GB RAM · coding total 102/126
- **NFS with TLS support (new feature)** – RHEL 10 now fully supports TLS for NFS RPC traffic via kernel TLS (kTLS). This encrypts all client‑server communication, improving security and compliance, while adding a modest encryption overhead that can affect throughput on high‑latency links.
- **IPv6 address parsing bug fix for `nfs://` URLs** – Previously, ReaR treated square brackets around IPv6 addresses as shell metacharacters, causing `nfs://[2001:db8::1]/…` URLs to fail and abort the backup/restore process. The fix now correctly parses bracketed IPv6 addresses, enabling reliable use of NFS mounts over IPv6 in backup scripts and other tooling.
- **FS‑Cache client‑side caching for NFS (new feature)** – RHEL 10 introduces FS‑Cache support for NFS via the `cachefiles` back‑end and `cachefilesd` daemon. When enabled, NFS data is cached locally on an ext3/4/XFS filesystem, reducing network traffic and server load and improving read performance for frequently accessed files. The trade‑off is that cache maintenance can add overhead and may slow the client if the cache is not properly sized or if many large files bypass the cache.- **NFS with TLS support (new feature)** – RHEL 10 now fully supports TLS for NFS RPC traffic, providing encrypted client‑server communication. This improves security and can affect performance due to the encryption overhead, but it enables secure NFS deployments that were previously limited to plaintext.
- **IPv6 address parsing bug fix for nfs:// URLs** – The kernel now correctly interprets IPv6 addresses enclosed in square brackets in `sshfs://` or `nfs://` URLs used in `BACKUP_URL`/`OUTPUT_URL`. This resolves a parsing error that caused ReaR to abort, allowing administrators to use IPv6 NFS endpoints without workarounds.
- **FS‑Cache client‑side caching for NFS (new feature)** – RHEL 10 introduces FS‑Cache support for NFS mounts, enabling local caching of NFS data via the `cachefiles` backend. This reduces network traffic, lowers server load, and can significantly improve read performance for frequently accessed files.- **NFS with TLS support (new feature)** – RHEL 10 now fully supports TLS for NFS RPC traffic, providing encrypted client‑server communication. This improves security and can affect performance due to the encryption overhead, but it enables secure NFS deployments that were previously limited to plaintext.
- **IPv6 address parsing bug fix for nfs:// URLs** – The kernel now correctly interprets IPv6 addresses enclosed in square brackets in `sshfs://` or `nfs://` URLs used in `BACKUP_URL`/`OUTPUT_URL`. This resolves a parsing error that caused ReaR to abort, allowing administrators to use IPv6 NFS endpoints without workarounds.
- **FS‑Cache client‑side caching for NFS (new feature)** – RHEL 10 introduces FS‑Cache support for NFS mounts, enabling local caching of NFS data via the `cachefiles` backend. This reduces network traffic, lowers server load, and can significantly improve read performance for frequently accessed files.Served via llama.cpp CUDA · 55.7 tok/s · 1.7 GB RAM · coding total 102/126
No failures.
Served via llama.cpp CUDA · 12.3 tok/s · 3.7 GB RAM · coding total 102/126
No failures.
Served via llama.cpp CUDA · 67.0 tok/s · 2.2 GB RAM · coding total 102/126
No failures.
Served via llama.cpp CUDA · 88.4 tok/s · 0.9 GB RAM · coding total 101/126
- **NFS with TLS support added** – NFS traffic can now be encrypted using TLS (via kernel TLS/kTLS), improving security for RPC communication between clients and servers.
- **Default `rsize` and `wsize` increased to 1,048,576 bytes** – Improves NFS read/write performance by allowing larger data transfers per operation.
- **FS-Cache (client-side caching) now supported for NFS** – Enables local caching of NFS content via `cachefilesd`, reducing network load and improving read performance (at potential cost of slight latency).
- **IPv6 addresses now supported in `nfs://` URLs** – Fixes parsing issue where square brackets around IPv6 addresses caused failures in tools like ReaR; improves compatibility with IPv6 NFS mounts.
- **FS-Cache requires specific backend filesystems (XFS, ext3, ext4)** – Ensures cache coherency and performance; administrators must configure `/var/cache/fscache/` on compatible filesystems.Served via llama.cpp CUDA · 12.4 tok/s · 3.6 GB RAM · coding total 99/126
No failures.
Served via llama.cpp CUDA · 13.7 tok/s · 1.2 GB RAM · coding total 98/126
No failures.
Served via llama.cpp CUDA · 66.5 tok/s · 2.2 GB RAM · coding total 96/126
No failures.
Served via llama.cpp CUDA · 67.4 tok/s · 0.9 GB RAM · coding total 96/126
* **NFS with TLS support added:** NFS traffic can now be encrypted using Transport Layer Security (TLS) via kernel TLS (kTLS). This matters because it enhances security by ensuring encrypted communication between clients and servers without requiring application-level changes.
* **IPv6 address support in NFS URLs fixed:** The ReaR tool now correctly interprets IPv6 addresses (enclosed in square brackets) in `nfs://` URLs. This matters because it resolves a bug that caused ReaR to abort when using IPv6 addresses in backup or output URLs.
* **NFS rsize and wsize maximums increased:** The maximum transfer size for NFS read and write operations is now 1,048,576 bytes (1 MiB). This matters because it allows larger data chunks to be transferred in a single operation, potentially improving throughput and reducing the number of round trips.
* **NFS client-side caching (FS-Cache) available:** The `cachefilesd` service can be used to cache NFS content on the client side. This matters because it reduces network and server load by satisfying read requests locally and enables efficient local data retrieval.
* **NFS mount options documentation updated:** The `vers` option is now documented as an alias for `nfsvers` for compatibility. This matters because it clarifies that both options function identically for NFS version selection.Served via llama.cpp CUDA · 11.8 tok/s · 3.5 GB RAM · coding total 96/126
No failures.
Served via llama.cpp CUDA · 22.3 tok/s · 3.7 GB RAM · coding total 93/126
No failures.
Served via llama.cpp CUDA · 68.1 tok/s · 0.9 GB RAM · coding total 90/126
No failures.
Served via llama.cpp CUDA · 48.3 tok/s · 1.2 GB RAM · coding total 88/126
No failures.
Served via llama.cpp CUDA · 36.1 tok/s · 0.9 GB RAM · coding total 83/126
No failures.
Served via llama.cpp CUDA · 8.9 tok/s · 1.4 GB RAM · coding total 81/126
- **Fixed parsing of IPv6 addresses in NFS URLs**: ReaR now correctly interprets IPv6 addresses enclosed in square brackets (e.g., `nfs://[2001:db8- **Fixed parsing of IPv6 addresses in NFS URLs**: ReaR now correctly interprets IPv6 addresses enclosed in square brackets (e.g., `nfs://[2001:db8Served via llama.cpp CUDA · 9.3 tok/s · 4.7 GB RAM · coding total 73/126
- **NFS with TLS support**: RHEL 10 offers NFS with Transport Layer Security (TLS), enhancing security by enabling encrypted communication between clients and servers. This feature relies on kernel TLS (kTLS), provided as a Technology Preview.
- **IPv6 address parsing fix**: A bug affecting the use of IPv6 addresses in sshfs:// and nfs:// URLs has been fixed. Now, square brackets around IPv6 addresses are correctly interpreted, allowing their use in these URL schemes.
- **NFS client-side caching**: RHEL 10 introduces client-side caching for NFS using FS-Cache, which can improve performance, reduce server load, and enable efficient local data retrieval. FS-Cache requires a compatible cache back-end file system (XFS, ext3, or ext4) and can be configured using the`/etc/cachefilesd.conf` file.<|END_OF_TURN_TOKEN|>- **NFS with TLS support**: RHEL 10 offers NFS with Transport Layer Security (TLS), enhancing security by enabling encrypted communication between clients and servers. This feature relies on kernel TLS (kTLS), provided as a Technology Preview.
- **IPv6 address parsing fix**: A bug affecting the use of IPv6 addresses in sshfs:// and nfs:// URLs has been fixed. Now, square brackets around IPv6 addresses are correctly interpreted, allowing their use in these URL schemes.
- **NFS client-side caching**: RHEL 10 introduces client-side caching for NFS using FS-Cache, which can improve performance, reduce server load, and enable efficient local data retrieval. FS-Cache requires a compatible cache back-end file system (XFS, ext3, or ext4) and can be configured using the`/etc/cachefilesd.conf` file.<|END_OF_TURN_TOKEN|>- **NFS with TLS support**: RHEL 10 offers NFS with Transport Layer Security (TLS), enhancing security by enabling encrypted communication between clients and servers. This feature relies on kernel TLS (kTLS), provided as a Technology Preview.
- **IPv6 address parsing fix**: A bug affecting the use of IPv6 addresses in sshfs:// and nfs:// URLs has been fixed. Now, square brackets around IPv6 addresses are correctly interpreted, allowing their use in these URL schemes.
- **NFS client-side caching**: RHEL 10 introduces client-side caching for NFS using FS-Cache, which can improve performance, reduce server load, and enable efficient local data retrieval. FS-Cache requires a compatible cache back-end file system (XFS, ext3, or ext4) and can be configured using the`/etc/cachefilesd.conf` file.<|END_OF_TURN_TOKEN|>