Intraductal papillary neoplasm of the bile duct (IPNB) represents biliary papillary tumors mainly growing and is considered to be of relatively low-grade malignancy. Here we report a case of IPNB in whom the poorly differentiated component deeply infiltrated the bile duct wall. A 77-year-old male had an invasive carcinoma of the bile duct 3 cm in size. He underwent right hemihepatectomy with combined resection of the extrahepatic bile duct. Papillary growing tumor was observed in the common bile duct and the right posterior Glisson’s pedicle was invaded. Histologic finding showed papillary adenocarcinoma in the surface layer superficially extending to the epithelium of the surrounding bile duct. In the subserosal layer, the tumor represented poorly differentiated adenocarcinoma. The tumor was diagnosed as invasive bile duct carcinoma arising from IPNB.