Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.4k views
in Technique[技术] by (71.8m points)

performance - Powershell Count lines extremely large file

I have a extremely large text file of size 250 GB that's given to us by a vendor. They also give us a control file that is supposed to have the number of lines in the large file. Sometimes there is a mismatch. How do I count lines in Powershell? I tried this command and it ran for more than half hour and was not done yet.

Get-content C:est.txt | Measure-Object –Line

(gc C:est.txt | Measure-object | select count).count

Any help is appreciated Thanks MR

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

If performance matters, avoid the use of cmdlets and the pipeline; use switch -File:

$count = 0
switch -File C:est.txt { default { ++$count } }

switch -File enumerates the lines of the specified file; condition default matches any line.


To give a sense of the performance difference:

# Create a sample file with 100,000 lines.
1..1e5 > tmp.txt
# Warm up the file cache
foreach ($line in [IO.File]::ReadLines("$pwd/tmp.txt")) { }

(Measure-Command { (Get-Content tmp.txt | Measure-Object).Count }).TotalSeconds

(Measure-Command { $count = 0; switch -File tmp.txt { default { ++$count } } }).TotalSeconds

Sample results from my Windows 10 / PSv5.1 machine:

1.3081307  # Get-Content + Measure-Object
0.1097513  # switch -File

That is, on my machine the switch -File command was about 12 times faster.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...