输入一个地址，就可以把那个网页中的链接提取出来，下面这段代码可以轻松实现，主要的是用到了正则表达式。查看例子
http://search.csdn.net/Expert/topic/2131/2131209.xml?temp=.4868585
GetUrl.aspx代码如下：
<%@ Page Language="vb" CodeBehind="GetUrl.aspx.vb" AutoEventWireup="false" Inherits="aspxWeb.GetUrl" %>
<html>
<head>
<META http-equiv="content-type" content="text/html; charset=gb2312">
</head>
<body>
<form id="Form1" method="post" runat="server">
<P>
<asp:Label id="Label1" runat="server"></asp:Label>
<asp:TextBox id="urlTextBox" runat="server" Width="336px">
http://lucky_elove.www1.dotnetplayground.com/
</asp:TextBox>
<asp:Button OnClick="scrapeButton_Click" id="scrapeButton" runat="server"></asp:Button>
</P>
<HR width="100%" SIZE="1">
<P>
<asp:Label id="TipResult" runat="server"></asp:Label>
<asp:TextBox id="resultLabel" runat="server" TextMode="MultiLine"
Width="100%" Height="400"></asp:TextBox>
</P>
</form>
</body>
</html>
后代码GetUrl.aspx.vb如下：
Imports System.IO
Imports System.Net
Imports System.Text
Imports System.Text.RegularExpressions
Imports System
Public Class GetUrl
Inherits System.Web.UI.Page
Protected WithEvents Label1 As System.Web.UI.WebControls.Label
Protected WithEvents urlTextBox As System.Web.UI.WebControls.TextBox
Protected WithEvents scrapeButton As System.Web.UI.WebControls.Button
Protected WithEvents TipResult As System.Web.UI.WebControls.Label
Protected WithEvents resultLabel As System.Web.UI.WebControls.TextBox
#Region " Web 窗体设计器生成的代码 "
'该调用是 Web 窗体设计器所必需的。
<System.Diagnostics.DebuggerStepThrough()> Private Sub InitializeComponent()
End Sub
Private Sub Page_Init(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Init
'CODEGEN: 此方法调用是 Web 窗体设计器所必需的
'不要使用代码编辑器修改它。
InitializeComponent()
End Sub
#End Region
Private Sub Page_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
'在此处放置初始化页的用户代码
Label1.Text = "请输入一个URL地址："
scrapeButton.Text = "分离Href链接"
End Sub
Private report As New StringBuilder()
Private webPage As String
Private countOfMatches As Int32
Public Sub scrapeButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs)
webPage = GrabUrl()
Dim myDelegate As New MatchEvaluator(AddressOf MatchHandler)
Dim linksExpression As New Regex( _
"\<a.+?href=['""](?!http\:\/\/)(?!mailto\:)(?>foundAnchor>[^'"">]+?)[^>]*?\>", _
RegexOptions.Multiline Or RegexOptions.IgnoreCase Or RegexOptions.IgnorePatternWhitespace)
Dim newWebPage As String = linksExpression.Replace(webPage, myDelegate)
TipResult.Text = "<h2>从 " & urlTextBox.Text & "分离出的Href链接</h2>" & _
"<b>找到并整理" & countOfMatches.ToString() & " 个链接</b><br><br>" & _
report.ToString().Replace(Environment.NewLine, "<br>")
TipResult.Text &= "<h2>整理过的页面</h2><script>window.document.title='抓取网页中的链接'</script>"
resultLabel.Text = newWebPage
End Sub
Public Function MatchHandler(ByVal m As Match) As String
Dim link As String = m.Groups("foundAnchor").Value
Dim rToL As New Regex("^", RegexOptions.Multiline Or RegexOptions.RightToLeft)
Dim col, row As Int32
Dim lineBegin As Int32 = rToL.Match(webPage, m.Index).Index
row = rToL.Matches(webPage, m.Index).Count
col = m.Index - lineBegin
report.AppendFormat( _
"Link <b>{0}</b>, fixed at row: {1}, col: {2}{3}", _
Server.HtmlEncode(m.Groups(0).Value), _
row, _
col, _
Environment.NewLine _
)
Dim newLink As String
If link.StartsWith("/") Then
newLink = link.Substring(1)
Else
newLink = link
End If
countOfMatches += 1
Return m.Groups(0).Value.Replace(link, newLink)
End Function
Private Function GrabUrl() As String
Dim wc As New WebClient()
Dim s As Stream = wc.OpenRead(urlTextBox.Text)
Dim sr As StreamReader = New StreamReader(s, System.Text.Encoding.Default)
GrabUrl = sr.ReadToEnd
s.Close()
wc.Dispose()
End Function
End Class
这段正则表达式了用来验证Email的：^[_.0-9a-z-]+@([0-9a-z][0-9a-z-]+.)+[a-z]{2,3}$
在这段正则表达式中，“+”表示前面的字符串连续出现一个或多个；“^”表示下一个字符串必须出现在开头，“$”表示前一个字符串必须出现在结尾；“.”也就是“.”，这里“”是转义符；“{2,3}”表示前面的字符串可以连续出现2-3次。“()”表示包含的内容必须同时出现在目标对象中。“[_.0-9a-z-]”表示包含在“_”、“.”、“-”、从a到z范围内的字母、从0到9范围内的数字中的任意字符；
这样一来，这个正则表达式可以这样翻译：
“下面的字符必须在开头（^）”、“该字符必须包含在“_”、“.”、“-”、从a到z范围内的字母、从0到9范围内的数字中（[_.0-9a-z-]）”、“前面这个字符至少出现一次（+）”、@、“该字符串由一个包含在从a到z范围内的一个字母、从0到9范围内的数字中的字符开头，后面跟至少一个包含在“-”、从a到z范围内任何一个字母、从0到9范围内任何一个数字中的字符，最后以.结束（([0-9a-z][0-9a-z-]+.)）”、“前面这个字符至少出现一次（+）”、“从a到z范围内的字母出现2-3次，并以它结束（[a-z]{2,3}$）”
表示匹配但不获取，如果不用，会多获取几个匹配，占用资源。
?<1>引用名称，即表示该获取可以用$1引用，
如果想更好的使用正则，可以看以下连接，希望对你有用
http://expert.csdn.net/Expert/TopicView1.asp?id=1410423
单看一眼，和href\s*=\s*("[^"]"|\S+)区别
只是这个如果有引号会同时匹配到，同时也有$1
上面只是获取""内的内容，复杂了点但有实用
http://search.csdn.net/Expert/topic/1450/1450366.xml?temp=.8075525
http://search.csdn.net/Expert/topic/1895/1895427.xml?temp=.2321894
例如：
当前网页为：http://www.cccc.com/aa/bb/cc/dd/ee.htm，网页为有如下代码：
（1）href="../../../df/gov.htm"
需要转为：href="http://www.cccc.com/aa/df/gov.htm"
（2）href="../../special_index.htm"
需要转为：href="http://www.cccc.com/aa/bb/spcial_index.htm"
(3)href="/index.htm" class='white'>频道主页</a>
需要转为：href="http://www.cccc.com/index.htm" class='white'>频道主页</a>
(4)<img src=/myexe/wind_chromeless_2.1.js>
需要转为：<img src=http://www.cccc.com/myexe/wind_chromeless_2.1.js>
(5)background-image: url(images/index/sameA.jpg)[css代码中]
需要转为：background-image: url(http://www.cccc.com/aa/bb/cc/dd/images/index/sameA.jpg)
(6)url="mailto:[email protected]"又不能转换。
现在比较头痛的问题是：网页并不都是很规范的。有的属性值用了引号（或单引号或双引号），还有的没有用引号，更有甚者，单双引号嵌套。但又必须保证不乱，否则就会乱掉。
1.首先确定出 http://www.cccc.com/aa/bb/cc/dd/ee.htm 的根 http://www.cccc.com
2.确定出 http://www.cccc.com/aa/bb/cc/dd/ee.htm 得目录 http://www.cccc.com/aa/bb/cc/dd
2.在所有像 href="/、src="/ 这样的前面加入 http://www.cccc.com
3.在所有 href="???.htm、src="???.jpg 前面加入 http://www.cccc.com/aa/bb/cc/dd/
4.像 href="../../../df/gov.htm" 计算出有多少个 ../ ，有3个，表示从 http://www.cccc.com/aa/bb/cc/dd 后退3层，先将 http://www.cccc.com/aa/bb/cc/dd 反向排列，找到第3个 / 的位置，从这个位置开始提取字符，将提取的字符再次反向，得到http://www.cccc.com/aa/，提取 href="../../../df/gov.htm" 中的第3个 / 后面所有字符和前面得到的 http://www.cccc.com/aa/ 组合。
5.像 ../ 和 ../../ 用 4 的办法同样计算。
6.像 url(???/??.jpg) 只要在 url( 后面插入 http://www.cccc.com/aa/bb/cc/dd 就行了。
用正则表达式
string pattern = @"(href\s*=\s*)|(src\s*=\s*)[""'](?<url>[^""']+)[""']";
Regex r = new Regex(pattern, RegexOptions.Compile | RegexOptions.IgnoreCase);
for(Match m = r.Match(YourHtmlPageString); m.Sucess; m = m.NextMatch())
{
string url = m.Result("${url}");
// 处理该URL
}
已经解决。
<%@ Page Language="VB" debug="true"%>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<script language="VB" runat="server">
Sub Page_load(sender as Object,E as EventArgs)
If IsPostBack=False Then
dim strUrl as string
strUrl=Request.QueryString("Url")
if strUrl="" then
strUrl=trim(Request.Params("Url"))
end if
strUrl=strUrl.TrimEnd("/")
' response.write(strUrl & "<br>")
if strUrl<>Nothing And strUrl.StartsWith("http://") then
Dim wc As New System.Net.WebClient()
Dim html As String = Encoding.default.GetString(wc.DownloadData(strUrl))
' Response.Write(html)
Dim strRegEx as String
strRegEx="\b(href|src|url|background)=((""|')?\s*([^\>\s]*?)\2?(\s)|([^>]*?>))"
html=RegExLinks(strRegEx,html,strUrl)
' strRegEx="\b(href|src|background)=(""|')?\s*([^\>\s]*?)\2?(\s)"
' html=RegExLinks(strRegEx,html,strUrl)
' strRegEx="\b(href|src|background)\s*=\s*(""|')?\s*([^>\s]*?)\2?\/?>"
' html=RegExLinks(strRegEx,html,strUrl)
Response.write(html)
end if
End If
End Sub
Function RegExLinks(ByVal strRegEx as string,ByVal html as string,ByVal strUrl as string)
dim arrLink() as String
dim firstquot,lastquot as string
dim strOldFullLink,strOldLink,strNewFullLink,strNewLink as String
dim strLink as String
dim strSpace as String
dim objRegEx as RegEx
Dim objMatch as Match
Dim objMatchCollection as MatchCollection
objRegEx=New RegEx(strRegEx,RegexOptions.IgnoreCase or RegexOptions.Multiline)
objMatchCollection=objRegEx.Matches(html)
For Each objMatch in objMatchCollection
strLink=objMatch.value
Erase arrLink
arrLink=strLink.split("=")
'如果链接中有http://www.domain.com/news.asp?date=200306&keyword=news&page=2等类似情况时，Ubound>=2，此时后面无空格，否则错误
if UBound(arrLink)<2 then
strSpace=" "
else
strSpace=""
end if
if arrLink(1).StartsWith("""") then
strOldFullLink=arrLink(1)
if arrLink(1).LastIndexOf("""")>1 then
if arrLink(1).EndsWith(">") then
arrLink(1)=arrLink(1).TrimEnd(">")
lastquot=""">"
else
lastquot=""""
end if
end if
strOldLink=arrLink(1).replace("""","")
firstquot=""""
strNewLink=DoLinks(strUrl,strOldLink)
strNewFullLink=firstquot & trim(strNewLink) & trim(lastquot)
' response.write("替换前:双引号" & strOldFullLink & "<br>")
' response.write("替换后:双引号<font color='red'>" & strNewFullLink & "</font><br>")
elseif arrLink(1).StartsWith("'") then
strOldFullLink=arrLink(1)
if arrLink(1).LastIndexOf("'")>1 then
if arrLink(1).EndsWith(">") then
arrLink(1)=arrLink(1).TrimEnd(">")
lastquot="'>"
else
lastquot="'"
end if
end if
strOldLink=arrLink(1).replace("'","")
firstquot="'"
strNewLink=DoLinks(strUrl,strOldLink)
strNewFullLink=firstquot & trim(strNewLink) & trim(lastquot)
' response.write("替换前:单" & strOldFullLink & "<br>")
' response.write("替换后:单<font color='red'>" & strNewFullLink & "</font><br>")
else
strOldFullLink=arrLink(0) & "=" & arrLink(1)
' strOldFullLink=arrLink(1)
strOldLink=arrLink(1)
strNewLink=DoLinks(strUrl,strOldLink)
strNewFullLink=arrLink(0) & "=" & trim(strNewLink)
' strNewFullLink=trim(strNewLink)
' response.write("前:无" & strOldFullLink & "<br>")
' response.write("后:无<font color='red'>" & strNewFullLink & "</font><br>")
end if
html=html.Replace(strOldFullLink,trim(strNewFullLink) & strSpace)
firstquot=nothing
lastquot=nothing
strOldFullLink=nothing
strNewFullLink=nothing
Next
RegExLinks=html
End Function
Function DoLinks(byVal strUrl as string,byVal strTempLink as string)
dim objRegExSite as RegEx
objRegExSite=New RegEx("http://[^/]+",RegexOptions.IgnoreCase)
dim strSite as string
strSite=trim(objRegExSite.Match(strUrl).value.ToString)
dim strLinkF as String
dim strUrlF as String
strUrlF=strUrl.Replace(strSite,"")
dim arrDir() as String
dim iDirLen as integer
if strUrlF.indexOf("/")>=0 then
arrDir=strUrlF.split("/")
iDirLen=arrDir.length
strUrlF=strUrlF.Replace(arrDir(iDirLen-1),"")
end if
dim k,j as Integer
dim objMatchColF as MatchCollection
dim objRegExF as RegEx
if strTempLink.ToLower.StartsWith("javascript:") or strTempLink.ToLower.StartsWith("mailto:") or strTempLink.ToLower.StartsWith("#") or _
strTempLink.ToLower.StartsWith("http://") or strTempLink.ToLower.StartsWith("www.") then
strLinkF=strTempLink
elseif strTempLink.StartsWith("../") then
objRegExF=New RegEx("\.\.\/")
objMatchColF=objRegExF.Matches(strTempLink)
j=objMatchColF.Count
'当下载网页链接的"../"个数+1大于该网页链接层数时，说明网页本身有误，则指向最底层链接。
if isArray(arrDir) then
if Ubound(arrDir)<j+1 then
j=Ubound(arrDir)-1
end if
for k=j-1 to 0 step -1
strUrlF=trim(strUrlF.Remove(strUrlF.LastIndexOf(arrDir(iDirLen-2-k)),len(arrDir(iDirLen-2-k))+1))
next
end if
dim strEnd as String
strEnd=trim(strTempLink.Replace("../",""))
strLinkF=strSite.subString(0,len(strSite)) & strUrlF & strEnd
elseif strTempLink.StartsWith("./") then
' http://www.southcn.com/news/china
' ./todaycn/200306260529.htm
strLinkF=strUrl & strTempLink.Replace("./","/")
elseif strTempLink.StartsWith("/") then
strLinkF=strSite & strTempLink
else
if strUrlF="" then
strUrlF="/"
end if
strLinkF=strSite & strUrlF & strTempLink
end if
DoLinks=strLinkF
End Function
</script>
<html>
<body>
</body>
go to
http://www.regexlib.com/search.aspx
enter "links" in the keyword textbox and click on Search button
or try
using System.Text.RegularExpressions;
string str = "............";
Regex re = new Regex(@"<a[^>]+href=\s*(?:'(?<href>[^']+)'|""(?<href>[^""]+)""|(?<href>[^>\s]+))\s*[^>]*>", RegexOptions.IgnoreCase | RegexOptions.Singleline);
MatchCollection mc = re.Matches(str);
Console.WriteLine(mc.Count);
foreach (Match m in mc)
Console.WriteLine(m.Groups["href"].Value);
}
|
本类阅读TOP10