BBC News Robots.txt(news.bbc.co.uk)2 points·by JKFSOM·10 месяцев назад·1 commentsnews.bbc.co.ukBBC News Robots.txthttps://news.bbc.co.uk/robots.txt1 commentsPost comment[–]JKFSOM·10 месяцев назадreplyThe BBC News robots.txt file includes an interesting selection of disallowed pages:- "Jailed refugee wins right to stay" (http://news.bbc.co.uk/1/hi/england/1678616.stm) - "Teacher's HIV sack claim rejected" (http://news.bbc.co.uk/1/hi/england/beds/bucks/herts/3663494....) - "Should the UK vote on the EU constitution?" subpage (http://news.bbc.co.uk/1/shared/spl/hi/pop_ups/05/uk_should_t...) - Retail students cash in their talent (http://news.bbc.co.uk/1/hi/education/3826037.stm)
- "Jailed refugee wins right to stay" (http://news.bbc.co.uk/1/hi/england/1678616.stm) - "Teacher's HIV sack claim rejected" (http://news.bbc.co.uk/1/hi/england/beds/bucks/herts/3663494....) - "Should the UK vote on the EU constitution?" subpage (http://news.bbc.co.uk/1/shared/spl/hi/pop_ups/05/uk_should_t...) - Retail students cash in their talent (http://news.bbc.co.uk/1/hi/education/3826037.stm)