How to Efficiently Select Random Rows from a Large PostgreSQL Table?
Jan 21, 2025 am 05:46 AM
Select random rows from large PostgreSQL table
When working with large data sets, selecting random rows can be a computationally intensive task. This article explores various methods for retrieving random rows from a table containing approximately 500 million rows, and discusses their performance and accuracy.
Method 1: Use RANDOM() and LIMIT
The first method involves using the RANDOM() function to generate random numbers and then using the LIMIT clause to filter the results to get the required number of rows.
SELECT * FROM table WHERE RANDOM() < 0.000002 LIMIT 1000;
This approach has the advantage of being easy to implement, but may be inefficient for large tables. Because of the LIMIT clause, the database must scan all rows of the table to pick random rows and discard the rest.
Method 2: Use ORDER BY RANDOM() and LIMIT
Another approach is to first sort the rows by the RANDOM() function and then use the LIMIT clause to get random rows.
SELECT * FROM table ORDER BY RANDOM() LIMIT 1000;
This method is similar to the first method, but the sorting guarantees more efficient selection of random rows. It reduces the number of scans required, making it a better choice for large tables. However, it is still not the best choice for tables with extremely large number of rows.
Efficient approach: use numeric ID columns and indexes
For tables with numeric ID columns and fewer gaps, a more efficient approach can be used. This involves generating random numbers within a range of IDs and using them to join with the table.
WITH params AS (
SELECT 1 AS min_id, -- 最小 ID <= 當(dāng)前最小 ID
5100000 AS id_span -- 四舍五入。(max_id - min_id + buffer)
)
SELECT *
FROM (
SELECT p.min_id + trunc(random() * p.id_span)::integer AS id
FROM params p, generate_series(1, 1100) g -- 1000 + buffer
GROUP BY 1 -- 去除重復(fù)項(xiàng)
) r
JOIN table USING (id)
LIMIT 1000;
This approach leverages index access to significantly reduce the number of scans required. It is ideal for tables with a large number of rows and few gaps in the ID column.
Considerations and Recommendations
The best way to select random rows depends on specific table characteristics and performance requirements. For small tables, the RANDOM() or ORDER BY RANDOM() methods may be sufficient. However, for large tables with numeric ID columns and few gaps, it is recommended to use the above optimization method for best performance.
It should be noted that due to the nature of pseudo-random number generation in computers, none of these methods can guarantee true randomness. However, they provide a practical way to obtain a random sample of rows from a large table with reasonable efficiency and accuracy.
The above is the detailed content of How to Efficiently Select Random Rows from a Large PostgreSQL Table?. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
What are the ACID properties of a MySQL transaction?
Jun 20, 2025 am 01:06 AM
MySQL transactions follow ACID characteristics to ensure the reliability and consistency of database transactions. First, atomicity ensures that transactions are executed as an indivisible whole, either all succeed or all fail to roll back. For example, withdrawals and deposits must be completed or not occur at the same time in the transfer operation; second, consistency ensures that transactions transition the database from one valid state to another, and maintains the correct data logic through mechanisms such as constraints and triggers; third, isolation controls the visibility of multiple transactions when concurrent execution, prevents dirty reading, non-repeatable reading and fantasy reading. MySQL supports ReadUncommitted and ReadCommi.
How to add the MySQL bin directory to the system PATH
Jul 01, 2025 am 01:39 AM
To add MySQL's bin directory to the system PATH, it needs to be configured according to the different operating systems. 1. Windows system: Find the bin folder in the MySQL installation directory (the default path is usually C:\ProgramFiles\MySQL\MySQLServerX.X\bin), right-click "This Computer" → "Properties" → "Advanced System Settings" → "Environment Variables", select Path in "System Variables" and edit it, add the MySQLbin path, save it and restart the command prompt and enter mysql--version verification; 2.macOS and Linux systems: Bash users edit ~/.bashrc or ~/.bash_
What are the transaction isolation levels in MySQL, and which is the default?
Jun 23, 2025 pm 03:05 PM
MySQL's default transaction isolation level is RepeatableRead, which prevents dirty reads and non-repeatable reads through MVCC and gap locks, and avoids phantom reading in most cases; other major levels include read uncommitted (ReadUncommitted), allowing dirty reads but the fastest performance, 1. Read Committed (ReadCommitted) ensures that the submitted data is read but may encounter non-repeatable reads and phantom readings, 2. RepeatableRead default level ensures that multiple reads within the transaction are consistent, 3. Serialization (Serializable) the highest level, prevents other transactions from modifying data through locks, ensuring data integrity but sacrificing performance;
Establishing secure remote connections to a MySQL server
Jul 04, 2025 am 01:44 AM
TosecurelyconnecttoaremoteMySQLserver,useSSHtunneling,configureMySQLforremoteaccess,setfirewallrules,andconsiderSSLencryption.First,establishanSSHtunnelwithssh-L3307:localhost:3306user@remote-server-Nandconnectviamysql-h127.0.0.1-P3307.Second,editMyS
Where does mysql workbench save connection information
Jun 26, 2025 am 05:23 AM
MySQLWorkbench stores connection information in the system configuration file. The specific path varies according to the operating system: 1. It is located in %APPDATA%\MySQL\Workbench\connections.xml in Windows system; 2. It is located in ~/Library/ApplicationSupport/MySQL/Workbench/connections.xml in macOS system; 3. It is usually located in ~/.mysql/workbench/connections.xml in Linux system or ~/.local/share/data/MySQL/Wor
What is the principle behind a database connection pool?
Jun 20, 2025 am 01:07 AM
Aconnectionpoolisacacheofdatabaseconnectionsthatarekeptopenandreusedtoimproveefficiency.Insteadofopeningandclosingconnectionsforeachrequest,theapplicationborrowsaconnectionfromthepool,usesit,andthenreturnsit,reducingoverheadandimprovingperformance.Co
Analyzing the MySQL Slow Query Log to Find Performance Bottlenecks
Jul 04, 2025 am 02:46 AM
Turn on MySQL slow query logs and analyze locationable performance issues. 1. Edit the configuration file or dynamically set slow_query_log and long_query_time; 2. The log contains key fields such as Query_time, Lock_time, Rows_examined to assist in judging efficiency bottlenecks; 3. Use mysqldumpslow or pt-query-digest tools to efficiently analyze logs; 4. Optimization suggestions include adding indexes, avoiding SELECT*, splitting complex queries, etc. For example, adding an index to user_id can significantly reduce the number of scanned rows and improve query efficiency.
Performing logical backups using mysqldump in MySQL
Jul 06, 2025 am 02:55 AM
mysqldump is a common tool for performing logical backups of MySQL databases. It generates SQL files containing CREATE and INSERT statements to rebuild the database. 1. It does not back up the original file, but converts the database structure and content into portable SQL commands; 2. It is suitable for small databases or selective recovery, and is not suitable for fast recovery of TB-level data; 3. Common options include --single-transaction, --databases, --all-databases, --routines, etc.; 4. Use mysql command to import during recovery, and can turn off foreign key checks to improve speed; 5. It is recommended to test backup regularly, use compression, and automatic adjustment.
