输入一个地址,就可以把那个网页中的链接提取出来,下面这段代码可以轻松实现,主要的是用到了正则表达式。查看例子 http://search.csdn.net/Expert/topic/2131/2131209.xml?temp=.4868585
GetUrl.aspx代码如下:
<%@ Page Language="vb" CodeBehind="GetUrl.aspx.vb" AutoEventWireup="false" Inherits="aspxWeb.GetUrl" %> <html> <head> <META http-equiv="content-type" content="text/html; charset=gb2312"> </head> <body> <form id="Form1" method="post" runat="server"> <P> <asp:Label id="Label1" runat="server"></asp:Label> <asp:TextBox id="urlTextBox" runat="server" Width="336px"> http://lucky_elove.www1.dotnetplayground.com/ </asp:TextBox> <asp:Button OnClick="scrapeButton_Click" id="scrapeButton" runat="server"></asp:Button> </P> <HR width="100%" SIZE="1"> <P> <asp:Label id="TipResult" runat="server"></asp:Label> <asp:TextBox id="resultLabel" runat="server" TextMode="MultiLine" Width="100%" Height="400"></asp:TextBox> </P> </form> </body> </html> 后代码GetUrl.aspx.vb如下:
Imports System.IO Imports System.Net Imports System.Text Imports System.Text.RegularExpressions Imports System
Public Class GetUrl Inherits System.Web.UI.Page Protected WithEvents Label1 As System.Web.UI.WebControls.Label Protected WithEvents urlTextBox As System.Web.UI.WebControls.TextBox Protected WithEvents scrapeButton As System.Web.UI.WebControls.Button Protected WithEvents TipResult As System.Web.UI.WebControls.Label Protected WithEvents resultLabel As System.Web.UI.WebControls.TextBox
#Region " Web 窗体设计器生成的代码 "
'该调用是 Web 窗体设计器所必需的。 <System.Diagnostics.DebuggerStepThrough()> Private Sub InitializeComponent()
End Sub
Private Sub Page_Init(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Init 'CODEGEN: 此方法调用是 Web 窗体设计器所必需的 '不要使用代码编辑器修改它。 InitializeComponent() End Sub
#End Region
Private Sub Page_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load '在此处放置初始化页的用户代码 Label1.Text = "请输入一个URL地址:" scrapeButton.Text = "分离Href链接" End Sub Private report As New StringBuilder() Private webPage As String Private countOfMatches As Int32
Public Sub scrapeButton_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) webPage = GrabUrl() Dim myDelegate As New MatchEvaluator(AddressOf MatchHandler)
Dim linksExpression As New Regex( _ "\<a.+?href=['""](?!http\:\/\/)(?!mailto\:)(?>foundAnchor>[^'"">]+?)[^>]*?\>", _ RegexOptions.Multiline Or RegexOptions.IgnoreCase Or RegexOptions.IgnorePatternWhitespace)
Dim newWebPage As String = linksExpression.Replace(webPage, myDelegate)
TipResult.Text = "<h2>从 " & urlTextBox.Text & "分离出的Href链接</h2>" & _ "<b>找到并整理" & countOfMatches.ToString() & " 个链接</b><br><br>" & _ report.ToString().Replace(Environment.NewLine, "<br>") TipResult.Text &= "<h2>整理过的页面</h2><script>window.document.title='抓取网页中的链接'</script>" resultLabel.Text = newWebPage End Sub Public Function MatchHandler(ByVal m As Match) As String Dim link As String = m.Groups("foundAnchor").Value Dim rToL As New Regex("^", RegexOptions.Multiline Or RegexOptions.RightToLeft) Dim col, row As Int32 Dim lineBegin As Int32 = rToL.Match(webPage, m.Index).Index
row = rToL.Matches(webPage, m.Index).Count col = m.Index - lineBegin
report.AppendFormat( _ "Link <b>{0}</b>, fixed at row: {1}, col: {2}{3}", _ Server.HtmlEncode(m.Groups(0).Value), _ row, _ col, _ Environment.NewLine _ ) Dim newLink As String If link.StartsWith("/") Then newLink = link.Substring(1) Else newLink = link End If
countOfMatches += 1 Return m.Groups(0).Value.Replace(link, newLink) End Function
Private Function GrabUrl() As String Dim wc As New WebClient() Dim s As Stream = wc.OpenRead(urlTextBox.Text) Dim sr As StreamReader = New StreamReader(s, System.Text.Encoding.Default) GrabUrl = sr.ReadToEnd s.Close() wc.Dispose() End Function
End Class
这段正则表达式了用来验证Email的:^[_.0-9a-z-]+@([0-9a-z][0-9a-z-]+.)+[a-z]{2,3}$ 在这段正则表达式中,“+”表示前面的字符串连续出现一个或多个;“^”表示下一个字符串必须出现在开头,“$”表示前一个字符串必须出现在结尾;“.”也就是“.”,这里“”是转义符;“{2,3}”表示前面的字符串可以连续出现2-3次。“()”表示包含的内容必须同时出现在目标对象中。“[_.0-9a-z-]”表示包含在“_”、“.”、“-”、从a到z范围内的字母、从0到9范围内的数字中的任意字符; 这样一来,这个正则表达式可以这样翻译: “下面的字符必须在开头(^)”、“该字符必须包含在“_”、“.”、“-”、从a到z范围内的字母、从0到9范围内的数字中([_.0-9a-z-])”、“前面这个字符至少出现一次(+)”、@、“该字符串由一个包含在从a到z范围内的一个字母、从0到9范围内的数字中的字符开头,后面跟至少一个包含在“-”、从a到z范围内任何一个字母、从0到9范围内任何一个数字中的字符,最后以.结束(([0-9a-z][0-9a-z-]+.))”、“前面这个字符至少出现一次(+)”、“从a到z范围内的字母出现2-3次,并以它结束([a-z]{2,3}$)”
表示匹配但不获取,如果不用,会多获取几个匹配,占用资源。 ?<1>引用名称,即表示该获取可以用$1引用, 如果想更好的使用正则,可以看以下连接,希望对你有用 http://expert.csdn.net/Expert/TopicView1.asp?id=1410423
单看一眼,和href\s*=\s*("[^"]"|\S+)区别 只是这个如果有引号会同时匹配到,同时也有$1 上面只是获取""内的内容,复杂了点但有实用
http://search.csdn.net/Expert/topic/1450/1450366.xml?temp=.8075525
http://search.csdn.net/Expert/topic/1895/1895427.xml?temp=.2321894 例如: 当前网页为:http://www.cccc.com/aa/bb/cc/dd/ee.htm,网页为有如下代码:
(1)href="../../../df/gov.htm" 需要转为:href="http://www.cccc.com/aa/df/gov.htm"
(2)href="../../special_index.htm" 需要转为:href="http://www.cccc.com/aa/bb/spcial_index.htm"
(3)href="/index.htm" class='white'>频道主页</a> 需要转为:href="http://www.cccc.com/index.htm" class='white'>频道主页</a>
(4)<img src=/myexe/wind_chromeless_2.1.js> 需要转为:<img src=http://www.cccc.com/myexe/wind_chromeless_2.1.js>
(5)background-image: url(images/index/sameA.jpg)[css代码中] 需要转为:background-image: url(http://www.cccc.com/aa/bb/cc/dd/images/index/sameA.jpg)
(6)url="mailto:[email protected]"又不能转换。
现在比较头痛的问题是:网页并不都是很规范的。有的属性值用了引号(或单引号或双引号),还有的没有用引号,更有甚者,单双引号嵌套。但又必须保证不乱,否则就会乱掉。
1.首先确定出 http://www.cccc.com/aa/bb/cc/dd/ee.htm 的根 http://www.cccc.com
2.确定出 http://www.cccc.com/aa/bb/cc/dd/ee.htm 得目录 http://www.cccc.com/aa/bb/cc/dd
2.在所有像 href="/、src="/ 这样的前面加入 http://www.cccc.com
3.在所有 href="???.htm、src="???.jpg 前面加入 http://www.cccc.com/aa/bb/cc/dd/
4.像 href="../../../df/gov.htm" 计算出有多少个 ../ ,有3个,表示从 http://www.cccc.com/aa/bb/cc/dd 后退3层,先将 http://www.cccc.com/aa/bb/cc/dd 反向排列,找到第3个 / 的位置,从这个位置开始提取字符,将提取的字符再次反向,得到http://www.cccc.com/aa/,提取 href="../../../df/gov.htm" 中的第3个 / 后面所有字符和前面得到的 http://www.cccc.com/aa/ 组合。
5.像 ../ 和 ../../ 用 4 的办法同样计算。
6.像 url(???/??.jpg) 只要在 url( 后面插入 http://www.cccc.com/aa/bb/cc/dd 就行了。
用正则表达式 string pattern = @"(href\s*=\s*)|(src\s*=\s*)[""'](?<url>[^""']+)[""']"; Regex r = new Regex(pattern, RegexOptions.Compile | RegexOptions.IgnoreCase); for(Match m = r.Match(YourHtmlPageString); m.Sucess; m = m.NextMatch()) { string url = m.Result("${url}"); // 处理该URL }
已经解决。 <%@ Page Language="VB" debug="true"%> <%@ Import Namespace="System.Net" %> <%@ Import Namespace="System.IO" %> <script language="VB" runat="server"> Sub Page_load(sender as Object,E as EventArgs) If IsPostBack=False Then dim strUrl as string strUrl=Request.QueryString("Url") if strUrl="" then strUrl=trim(Request.Params("Url")) end if strUrl=strUrl.TrimEnd("/") ' response.write(strUrl & "<br>") if strUrl<>Nothing And strUrl.StartsWith("http://") then Dim wc As New System.Net.WebClient() Dim html As String = Encoding.default.GetString(wc.DownloadData(strUrl)) ' Response.Write(html) Dim strRegEx as String strRegEx="\b(href|src|url|background)=((""|')?\s*([^\>\s]*?)\2?(\s)|([^>]*?>))" html=RegExLinks(strRegEx,html,strUrl) ' strRegEx="\b(href|src|background)=(""|')?\s*([^\>\s]*?)\2?(\s)" ' html=RegExLinks(strRegEx,html,strUrl) ' strRegEx="\b(href|src|background)\s*=\s*(""|')?\s*([^>\s]*?)\2?\/?>" ' html=RegExLinks(strRegEx,html,strUrl) Response.write(html)
end if End If End Sub
Function RegExLinks(ByVal strRegEx as string,ByVal html as string,ByVal strUrl as string) dim arrLink() as String dim firstquot,lastquot as string dim strOldFullLink,strOldLink,strNewFullLink,strNewLink as String dim strLink as String dim strSpace as String dim objRegEx as RegEx Dim objMatch as Match Dim objMatchCollection as MatchCollection objRegEx=New RegEx(strRegEx,RegexOptions.IgnoreCase or RegexOptions.Multiline) objMatchCollection=objRegEx.Matches(html) For Each objMatch in objMatchCollection strLink=objMatch.value Erase arrLink arrLink=strLink.split("=") '如果链接中有http://www.domain.com/news.asp?date=200306&keyword=news&page=2等类似情况时,Ubound>=2,此时后面无空格,否则错误 if UBound(arrLink)<2 then strSpace=" " else strSpace="" end if if arrLink(1).StartsWith("""") then strOldFullLink=arrLink(1) if arrLink(1).LastIndexOf("""")>1 then if arrLink(1).EndsWith(">") then arrLink(1)=arrLink(1).TrimEnd(">") lastquot=""">" else lastquot="""" end if end if strOldLink=arrLink(1).replace("""","") firstquot="""" strNewLink=DoLinks(strUrl,strOldLink) strNewFullLink=firstquot & trim(strNewLink) & trim(lastquot) ' response.write("替换前:双引号" & strOldFullLink & "<br>") ' response.write("替换后:双引号<font color='red'>" & strNewFullLink & "</font><br>") elseif arrLink(1).StartsWith("'") then strOldFullLink=arrLink(1) if arrLink(1).LastIndexOf("'")>1 then if arrLink(1).EndsWith(">") then arrLink(1)=arrLink(1).TrimEnd(">") lastquot="'>" else lastquot="'" end if end if strOldLink=arrLink(1).replace("'","") firstquot="'" strNewLink=DoLinks(strUrl,strOldLink) strNewFullLink=firstquot & trim(strNewLink) & trim(lastquot) ' response.write("替换前:单" & strOldFullLink & "<br>") ' response.write("替换后:单<font color='red'>" & strNewFullLink & "</font><br>") else strOldFullLink=arrLink(0) & "=" & arrLink(1) ' strOldFullLink=arrLink(1) strOldLink=arrLink(1) strNewLink=DoLinks(strUrl,strOldLink) strNewFullLink=arrLink(0) & "=" & trim(strNewLink) ' strNewFullLink=trim(strNewLink) ' response.write("前:无" & strOldFullLink & "<br>") ' response.write("后:无<font color='red'>" & strNewFullLink & "</font><br>") end if html=html.Replace(strOldFullLink,trim(strNewFullLink) & strSpace)
firstquot=nothing lastquot=nothing strOldFullLink=nothing strNewFullLink=nothing Next RegExLinks=html End Function
Function DoLinks(byVal strUrl as string,byVal strTempLink as string) dim objRegExSite as RegEx objRegExSite=New RegEx("http://[^/]+",RegexOptions.IgnoreCase) dim strSite as string strSite=trim(objRegExSite.Match(strUrl).value.ToString) dim strLinkF as String dim strUrlF as String strUrlF=strUrl.Replace(strSite,"") dim arrDir() as String dim iDirLen as integer if strUrlF.indexOf("/")>=0 then arrDir=strUrlF.split("/") iDirLen=arrDir.length strUrlF=strUrlF.Replace(arrDir(iDirLen-1),"") end if
dim k,j as Integer dim objMatchColF as MatchCollection dim objRegExF as RegEx if strTempLink.ToLower.StartsWith("javascript:") or strTempLink.ToLower.StartsWith("mailto:") or strTempLink.ToLower.StartsWith("#") or _ strTempLink.ToLower.StartsWith("http://") or strTempLink.ToLower.StartsWith("www.") then strLinkF=strTempLink elseif strTempLink.StartsWith("../") then objRegExF=New RegEx("\.\.\/") objMatchColF=objRegExF.Matches(strTempLink) j=objMatchColF.Count '当下载网页链接的"../"个数+1大于该网页链接层数时,说明网页本身有误,则指向最底层链接。 if isArray(arrDir) then if Ubound(arrDir)<j+1 then j=Ubound(arrDir)-1 end if for k=j-1 to 0 step -1 strUrlF=trim(strUrlF.Remove(strUrlF.LastIndexOf(arrDir(iDirLen-2-k)),len(arrDir(iDirLen-2-k))+1)) next end if dim strEnd as String strEnd=trim(strTempLink.Replace("../","")) strLinkF=strSite.subString(0,len(strSite)) & strUrlF & strEnd elseif strTempLink.StartsWith("./") then ' http://www.southcn.com/news/china ' ./todaycn/200306260529.htm strLinkF=strUrl & strTempLink.Replace("./","/") elseif strTempLink.StartsWith("/") then strLinkF=strSite & strTempLink else if strUrlF="" then strUrlF="/" end if strLinkF=strSite & strUrlF & strTempLink end if DoLinks=strLinkF End Function
</script> <html> <body> </body>
go to http://www.regexlib.com/search.aspx
enter "links" in the keyword textbox and click on Search button
or try
using System.Text.RegularExpressions;
string str = "............";
Regex re = new Regex(@"<a[^>]+href=\s*(?:'(?<href>[^']+)'|""(?<href>[^""]+)""|(?<href>[^>\s]+))\s*[^>]*>", RegexOptions.IgnoreCase | RegexOptions.Singleline);
MatchCollection mc = re.Matches(str); Console.WriteLine(mc.Count); foreach (Match m in mc) Console.WriteLine(m.Groups["href"].Value); }

|